{"id":180532,"date":"2026-07-07T18:39:12","date_gmt":"2026-07-07T18:39:12","guid":{"rendered":"http:\/\/ktromedia.com\/?p=180532"},"modified":"2026-07-07T18:39:12","modified_gmt":"2026-07-07T18:39:12","slug":"the-complete-guide-to-tool-selection-in-ai-agents","status":"publish","type":"post","link":"https:\/\/ktromedia.com\/?p=180532","title":{"rendered":"The Complete Guide to Tool Selection in AI Agents"},"content":{"rendered":"<div id=\"\">\n<p>In this article, you will learn why agent accuracy degrades as a tool catalog grows, and six practical techniques for keeping tool selection accurate and efficient at scale.<\/p>\n<p>Topics we will cover include:<\/p>\n<ul>\n<li>Why adding more tools to an agent causes tool hallucination and accuracy loss, not just slower responses.<\/li>\n<li>How gating, retrieval, routing, and planning each narrow down what the model sees before it has to choose a tool.<\/li>\n<li>How to build fallback logic and a benchmark harness so you can measure whether any of these fixes actually worked.<\/li>\n<\/ul>\n<p>None of this requires a bigger model, just a smarter view of what the model sees before it acts.<\/p>\n<h2>Introduction<\/h2>\n<p>You build an agent with five tools. It works flawlessly in the demo. Three months later, it has 40 file operations, CRM access, Slack, a calendar, and three different search APIs you bolted on for different teams. The same agent that nailed every demo now calls the wrong tool, hallucinates parameters borrowed from a different tool\u2019s schema, or stalls mid-task waiting on a call that should never have been made.<\/p>\n<p>Nothing about the model changed. The tool list did. This is not an edge case you\u2019ll eventually run into. It\u2019s the default trajectory of every agent that ships and then grows. Research analyzing MCP tool descriptions across the ecosystem has found that a high number contain at least one quality issue, and production benchmarks show agent accuracy degrading measurably once tool counts pass roughly 10 to 15. The <a href=\"https:\/\/arxiv.org\/abs\/2505.03275\" target=\"_blank\">RAG-MCP paper<\/a>, published in May 2025, put hard numbers on the fix: retrieval-based tool selection more than tripled tool selection accuracy from 13.62% to 43.13% while cutting prompt tokens by over half on the same benchmark tasks.<\/p>\n<p>Tool selection isn\u2019t a minor implementation detail you patch later. It\u2019s the architectural decision that determines whether an agent survives contact with a real tool catalog. This guide covers six techniques that solve it, in the order you\u2019d actually deploy them: gating, retrieval, routing, planning, fallback logic, and the benchmark that tells you whether any of it worked.<\/p>\n<h2>Why Tool Selection Breaks at Scale<\/h2>\n<p>Every tool definition \u2014 its name, description, and parameter schema \u2014 gets sent to the model on every single request, whether that tool gets used or not. With 50-plus tools, this can consume 5 to 7% of the model\u2019s context before the user\u2019s actual message arrives, crowding out the conversation history and reasoning space the task actually needs.<\/p>\n<p>The \u201c<strong>lost in the middle<\/strong>\u201d effect compounds this. Models recall information at the start and end of a context window far more reliably than information buried in the middle. With dozens of near-identical tool definitions stacked in sequence, the one tool that\u2019s actually right for the job often sits exactly in that dead zone, overlooked not because the model can\u2019t reason about it, but because attention is structurally pulled elsewhere.<\/p>\n<p>The second failure mode is worse: tool hallucination. When an LLM\u2019s attention spreads across too many similar-sounding tools, it either invents tool names that don\u2019t exist or calls the correct tool while filling in arguments borrowed from a different tool\u2019s schema. This is a hard failure. There\u2019s no \u201c<strong>slightly wrong<\/strong>\u201d way to call a nonexistent function.<\/p>\n<p>OpenAI documents a <a href=\"https:\/\/community.openai.com\/t\/maximum-amount-of-tools-for-the-bot-to-use\/665720\" target=\"_blank\">hard ceiling of 128 tools per agent<\/a>, but real degradation shows up well before that limit; most production teams see accuracy drop noticeably once they cross 15 to 20 tools in active rotation. The fix isn\u2019t a bigger context window. It\u2019s controlling what the model sees in the first place.<\/p>\n<h2>Gating: Deciding Whether a Tool Is Needed at All<\/h2>\n<p>Before you optimize which tool to pick, ask a cheaper question first: does this turn need a tool at all? A meaningful fraction of agent turns are purely conversational: \u201c<strong>thanks<\/strong>,\u201d \u201c<strong>what do you mean by that<\/strong>,\u201d a follow-up clarification. Running full retrieval and tool-selection reasoning on every single turn means paying the full agentic overhead even when the answer is \u201c<strong>no tool needed<\/strong>.\u201d<\/p>\n<p>A gate is a fast, cheap classifier \u2014 sometimes a small model call, sometimes just pattern matching \u2014 that runs before anything expensive does.<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b5533787976579\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\n# gate.py&#13;<br \/>\n# Prerequisites: none beyond Python&#8217;s standard library (re)&#13;<br \/>\n# Run: python gate.py&#13;<br \/>\n&#13;<br \/>\nimport re&#13;<br \/>\n&#13;<br \/>\nCONVERSATIONAL_PATTERNS = [&#13;<br \/>\n    r&#8221;^\\s*(thanks|thank you|thx|ok|okay|cool|got it|sounds good|sure|great)\\b&#8221;,&#13;<br \/>\n    r&#8221;^\\s*(hi|hello|hey|good morning|good evening)\\b&#8221;,&#13;<br \/>\n    r&#8221;^\\s*what do you mean\\b&#8221;,&#13;<br \/>\n    r&#8221;^\\s*can you (clarify|explain that)\\b&#8221;,&#13;<br \/>\n]&#13;<br \/>\n&#13;<br \/>\nACTION_KEYWORDS = [&#13;<br \/>\n    &#8220;send&#8221;, &#8220;create&#8221;, &#8220;search&#8221;, &#8220;find&#8221;, &#8220;look up&#8221;, &#8220;schedule&#8221;, &#8220;book&#8221;,&#13;<br \/>\n    &#8220;read&#8221;, &#8220;write&#8221;, &#8220;query&#8221;, &#8220;summarize&#8221;, &#8220;translate&#8221;, &#8220;check&#8221;,&#13;<br \/>\n]&#13;<br \/>\n&#13;<br \/>\ndef gate(query: str) -&gt; dict:&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    Cheap pre-filter that decides whether the full tool-selection pipeline&#13;<br \/>\n    needs to run at all. Short-circuits conversational turns before&#13;<br \/>\n    retrieval, routing, or planning ever fires.&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    q_lower = query.strip().lower()&#13;<br \/>\n&#13;<br \/>\n    # Tier 1: regex match against known conversational patterns &#8212; near-zero cost&#13;<br \/>\n    for pattern in CONVERSATIONAL_PATTERNS:&#13;<br \/>\n        if re.match(pattern, q_lower):&#13;<br \/>\n            return {&#8220;tool_needed&#8221;: False, &#8220;reason&#8221;: &#8220;conversational_pattern&#8221;, &#8220;tier&#8221;: 1}&#13;<br \/>\n&#13;<br \/>\n    # Tier 2: if there&#8217;s no action verb and the message is short, likely no tool needed&#13;<br \/>\n    has_action_keyword = any(kw in q_lower for kw in ACTION_KEYWORDS)&#13;<br \/>\n    if not has_action_keyword and len(q_lower.split())  tool_needed={result[&#8216;tool_needed&#8217;]} ({result[&#8216;reason&#8217;]})&#8221;)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\" style=\"font-size: 12px !important; line-height: 15px !important;\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<p>47<\/p>\n<p>48<\/p>\n<p>49<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-p\"># gate.py<\/span><\/p>\n<p><span class=\"crayon-p\"># Prerequisites: none beyond Python&#8217;s standard library (re)<\/span><\/p>\n<p><span class=\"crayon-p\"># Run: python gate.py<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">re<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">CONVERSATIONAL_PATTERNS<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-i\">r<\/span><span class=\"crayon-s\">&#8220;^\\s*(thanks|thank you|thx|ok|okay|cool|got it|sounds good|sure|great)\\b&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-i\">r<\/span><span class=\"crayon-s\">&#8220;^\\s*(hi|hello|hey|good morning|good evening)\\b&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-i\">r<\/span><span class=\"crayon-s\">&#8220;^\\s*what do you mean\\b&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-i\">r<\/span><span class=\"crayon-s\">&#8220;^\\s*can you (clarify|explain that)\\b&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">ACTION_KEYWORDS<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;send&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;create&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;search&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;find&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;look up&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;schedule&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;book&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;read&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;write&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;query&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;summarize&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;translate&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;check&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">gate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Cheap pre-filter that decides whether the full tool-selection pipeline<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0needs to run at all. Short-circuits conversational turns before<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0retrieval, routing, or planning ever fires.<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">q_lower<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">strip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">lower<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># Tier 1: regex match against known conversational patterns &#8212; near-zero cost<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">pattern <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">CONVERSATIONAL_PATTERNS<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">re<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">match<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">pattern<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">q_lower<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;tool_needed&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">False<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;reason&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;conversational_pattern&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;tier&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># Tier 2: if there&#8217;s no action verb and the message is short, likely no tool needed<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">has_action_keyword<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">any<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">kw <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">q_lower <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">kw <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ACTION_KEYWORDS<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">not<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">has_action_keyword <\/span><span class=\"crayon-st\">and<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">q_lower<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">split<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\"><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">5<\/span><span class=\"crayon-o\">:<\/span><\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;tool_needed&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">False<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;reason&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;short_with_no_action_keyword&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;tier&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;tool_needed&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">True<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;reason&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;action_keyword_or_long_query&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;tier&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">__name__<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;__main__&#8221;<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">test_queries<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;thanks!&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;What&#8217;s the weather like in Lagos today?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;ok&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;Can you send an email to the sales team about the delay?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">q<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">test_queries<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">gate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">q<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;&#8216;{q}&#8217; -&gt; tool_needed={result[&#8216;tool_needed&#8217;]} ({result[&#8216;reason&#8217;]})&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>How to run (no dependencies required):<\/p>\n<p>This costs almost nothing and catches a meaningful share of turns before they reach the expensive part of the pipeline. The threshold for \u201cis this worth building\u201d is low: if even 20\u201330% of your turns are conversational, gating pays for itself immediately in both latency and token cost.<\/p>\n<h2>Retrieval-Based Tool Selection<\/h2>\n<p>This is the technique with the strongest published evidence behind it. Instead of sending every tool definition on every call, you index tool descriptions in a vector store, embed the incoming query, retrieve only the top-K most relevant tools, and send just those to the model.<\/p>\n<p>The <a href=\"https:\/\/arxiv.org\/abs\/2505.03275\" target=\"_blank\">RAG-MCP framework<\/a> is the reference implementation of this idea, using semantic retrieval to identify the most relevant MCP tools for a query before the LLM ever sees the full catalog. The reported numbers are not subtle: tool selection accuracy rose from 13.62% with the full catalog exposed to 43.13% with retrieval-filtered selection, more than tripling accuracy, while cutting prompt tokens by over 50% on the same benchmark tasks.<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b5548652383563\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\n# retriever.py&#13;<br \/>\n# Prerequisites: pip install sentence-transformers faiss-cpu numpy&#13;<br \/>\n# Run: python retriever.py&#13;<br \/>\n&#13;<br \/>\nimport numpy as np&#13;<br \/>\nfrom sentence_transformers import SentenceTransformer&#13;<br \/>\nimport faiss&#13;<br \/>\n&#13;<br \/>\nTOOL_CATALOG = [&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;search_web&#8221;, &#8220;description&#8221;: &#8220;Search the web for current information on any topic&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;read_file&#8221;, &#8220;description&#8221;: &#8220;Read the contents of a file given its path&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;write_file&#8221;, &#8220;description&#8221;: &#8220;Write or overwrite content to a file at a given path&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;send_email&#8221;, &#8220;description&#8221;: &#8220;Send an email to a recipient with subject and body&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;create_calendar_event&#8221;, &#8220;description&#8221;: &#8220;Create a new calendar event with a title, date, and time&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;query_database&#8221;, &#8220;description&#8221;: &#8220;Run a SQL query against the company database&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;list_github_issues&#8221;, &#8220;description&#8221;: &#8220;List open issues in a GitHub repository&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;create_github_pr&#8221;, &#8220;description&#8221;: &#8220;Create a pull request on a GitHub repository&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;send_slack_message&#8221;, &#8220;description&#8221;: &#8220;Send a message to a Slack channel or user&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;get_weather&#8221;, &#8220;description&#8221;: &#8220;Get current weather conditions for a city&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;translate_text&#8221;, &#8220;description&#8221;: &#8220;Translate text from one language to another&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;summarize_document&#8221;, &#8220;description&#8221;: &#8220;Summarize a long document into key points&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;lookup_stock_price&#8221;, &#8220;description&#8221;: &#8220;Get the current stock price for a ticker symbol&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;book_flight&#8221;, &#8220;description&#8221;: &#8220;Search and book a flight between two cities&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;create_invoice&#8221;, &#8220;description&#8221;: &#8220;Generate an invoice for a customer with line items&#8221;},&#13;<br \/>\n]&#13;<br \/>\n&#13;<br \/>\nclass ToolRetriever:&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    Embeds tool descriptions once at startup and indexes them in FAISS.&#13;<br \/>\n    At runtime, embeds the incoming query and returns only the top-K&#13;<br \/>\n    most relevant tools &#8212; not the full catalog.&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    def __init__(self, tools: list[dict], model_name: str = &#8220;all-MiniLM-L6-v2&#8243;):&#13;<br \/>\n        self.tools = tools&#13;<br \/>\n        self.model = SentenceTransformer(model_name)&#13;<br \/>\n        descriptions = [f&#8221;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221; for t in tools]&#13;<br \/>\n        embeddings = self.model.encode(descriptions, normalize_embeddings=True)&#13;<br \/>\n        # IndexFlatIP = inner product search, which equals cosine similarity&#13;<br \/>\n        # when vectors are normalized &#8212; the standard setup for this use case.&#13;<br \/>\n        self.index = faiss.IndexFlatIP(embeddings.shape[1])&#13;<br \/>\n        self.index.add(np.array(embeddings, dtype=np.float32))&#13;<br \/>\n&#13;<br \/>\n    def retrieve(self, query: str, top_k: int = 3) -&gt; list[dict]:&#13;<br \/>\n        query_emb = self.model.encode([query], normalize_embeddings=True)&#13;<br \/>\n        scores, indices = self.index.search(np.array(query_emb, dtype=np.float32), top_k)&#13;<br \/>\n        return [&#13;<br \/>\n            {**self.tools[idx], &#8220;score&#8221;: float(score)}&#13;<br \/>\n            for score, idx in zip(scores[0], indices[0])&#13;<br \/>\n        ]&#13;<br \/>\n&#13;<br \/>\n&#13;<br \/>\nif __name__ == &#8220;__main__&#8221;:&#13;<br \/>\n    retriever = ToolRetriever(TOOL_CATALOG)&#13;<br \/>\n&#13;<br \/>\n    queries = [&#13;<br \/>\n        &#8220;What&#8217;s the weather like in Lagos today?&#8221;,&#13;<br \/>\n        &#8220;Can you check if there are any open bugs in our repo?&#8221;,&#13;<br \/>\n        &#8220;Send a message to the engineering channel about the deploy&#8221;,&#13;<br \/>\n    ]&#13;<br \/>\n    for q in queries:&#13;<br \/>\n        results = retriever.retrieve(q, top_k=3)&#13;<br \/>\n        print(f&#8221;\\nQuery: &#8216;{q}'&#8221;)&#13;<br \/>\n        for r in results:&#13;<br \/>\n            print(f&#8221;  {r[&#8216;name&#8217;]} (score={r[&#8216;score&#8217;]:.3f})&#8221;)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\" style=\"font-size: 12px !important; line-height: 15px !important;\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<p>47<\/p>\n<p>48<\/p>\n<p>49<\/p>\n<p>50<\/p>\n<p>51<\/p>\n<p>52<\/p>\n<p>53<\/p>\n<p>54<\/p>\n<p>55<\/p>\n<p>56<\/p>\n<p>57<\/p>\n<p>58<\/p>\n<p>59<\/p>\n<p>60<\/p>\n<p>61<\/p>\n<p>62<\/p>\n<p>63<\/p>\n<p>64<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-p\"># retriever.py<\/span><\/p>\n<p><span class=\"crayon-p\"># Prerequisites: pip install sentence-transformers faiss-cpu numpy<\/span><\/p>\n<p><span class=\"crayon-p\"># Run: python retriever.py<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">np<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">sentence_transformers <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">SentenceTransformer<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">faiss<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">TOOL_CATALOG<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;search_web&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Search the web for current information on any topic&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;read_file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Read the contents of a file given its path&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;write_file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Write or overwrite content to a file at a given path&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_email&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Send an email to a recipient with subject and body&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;create_calendar_event&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Create a new calendar event with a title, date, and time&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;query_database&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Run a SQL query against the company database&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;list_github_issues&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;List open issues in a GitHub repository&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;create_github_pr&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Create a pull request on a GitHub repository&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_slack_message&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Send a message to a Slack channel or user&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;get_weather&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Get current weather conditions for a city&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;translate_text&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Translate text from one language to another&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;summarize_document&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Summarize a long document into key points&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;lookup_stock_price&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Get the current stock price for a ticker symbol&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;book_flight&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Search and book a flight between two cities&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;create_invoice&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Generate an invoice for a customer with line items&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">ToolRetriever<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Embeds tool descriptions once at startup and indexes them in FAISS.<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0At runtime, embeds the incoming query and returns only the top-K<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0most relevant tools &#8212; not the full catalog.<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">__init__<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model_name<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;all-MiniLM-L6-v2&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">tools<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">SentenceTransformer<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">model_name<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">descriptions<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">t<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">embeddings<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">encode<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">descriptions<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">normalize_embeddings<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-t\">True<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># IndexFlatIP = inner product search, which equals cosine similarity<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># when vectors are normalized &#8212; the standard setup for this use case.<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">index<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">faiss<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">IndexFlatIP<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">embeddings<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">index<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">add<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-t\">array<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">embeddings<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dtype<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">float32<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">retrieve<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">int<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">query_emb<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">encode<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">normalize_embeddings<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-t\">True<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">scores<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">indices<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">index<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">search<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-t\">array<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">query_emb<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dtype<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">float32<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">idx<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;score&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">score<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">idx <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">zip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">scores<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">indices<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">__name__<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;__main__&#8221;<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">retriever<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">ToolRetriever<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">TOOL_CATALOG<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">queries<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;What&#8217;s the weather like in Lagos today?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;Can you check if there are any open bugs in our repo?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;Send a message to the engineering channel about the deploy&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">q<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">queries<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">results<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">retriever<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">retrieve<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">q<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;\\nQuery: &#8216;{q}'&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">r<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">results<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0{r[&#8216;name&#8217;]} (score={r[&#8216;score&#8217;]:.3f})&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>How to run:<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b5551565633803\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\npip install sentence-transformers faiss-cpu numpy&#13;<br \/>\npython retriever.py<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-e\">pip <\/span><span class=\"crayon-e\">install <\/span><span class=\"crayon-v\">sentence<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-e\">transformers <\/span><span class=\"crayon-v\">faiss<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-e\">cpu <\/span><span class=\"crayon-e\">numpy<\/span><\/p>\n<p><span class=\"crayon-e\">python <\/span><span class=\"crayon-v\">retriever<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">py<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>Only the top-3 tools out of a 15-tool catalog get sent to the model per query, an 80% reduction in tool definitions on every call, and the accuracy lift compounds because the model is now choosing between a handful of genuinely relevant candidates instead of scanning past a dozen near-misses.<\/p>\n<h2>Semantic Routing<\/h2>\n<p>Routing is retrieval\u2019s lighter cousin, and it fits a different shape of problem. Retrieval answers \u201c<strong>which specific tool<\/strong>\u201d out of a flat list. Routing answers \u201c<strong>which toolbox<\/strong>\u201d \u2014 useful when your tools cluster naturally into categories (data, communication, scheduling) and you want to load only the relevant category\u2019s tools rather than re-ranking the entire catalog every time.<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b5556199065581\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\n# router.py&#13;<br \/>\n# Prerequisites: pip install scikit-learn numpy&#13;<br \/>\n# Run: python router.py&#13;<br \/>\n&#13;<br \/>\nimport numpy as np&#13;<br \/>\nfrom sklearn.feature_extraction.text import TfidfVectorizer&#13;<br \/>\nfrom sklearn.metrics.pairwise import cosine_similarity&#13;<br \/>\n&#13;<br \/>\nCATEGORIES = {&#13;<br \/>\n    &#8220;data&#8221;:          [&#8220;query the database&#8221;, &#8220;read a file&#8221;, &#8220;write data to storage&#8221;, &#8220;run a SQL query&#8221;],&#13;<br \/>\n    &#8220;communication&#8221;: [&#8220;send an email&#8221;, &#8220;post a slack message&#8221;, &#8220;notify the team&#8221;, &#8220;send a message&#8221;],&#13;<br \/>\n    &#8220;scheduling&#8221;:    [&#8220;create a calendar event&#8221;, &#8220;book a meeting&#8221;, &#8220;schedule an appointment&#8221;],&#13;<br \/>\n}&#13;<br \/>\n&#13;<br \/>\nclass SemanticRouter:&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    Routes a query to a tool category using similarity against category&#13;<br \/>\n    centroid embeddings. Falls back to &#8216;general&#8217; when no category clears&#13;<br \/>\n    the confidence threshold &#8212; never guesses on a low-confidence match.&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    def __init__(self, categories: dict[str, list[str]], confidence_threshold: float = 0.15):&#13;<br \/>\n        self.threshold = confidence_threshold&#13;<br \/>\n        self.vectorizer = TfidfVectorizer()&#13;<br \/>\n        all_examples = [ex for exs in categories.values() for ex in exs]&#13;<br \/>\n        self.vectorizer.fit(all_examples)&#13;<br \/>\n&#13;<br \/>\n        # Build one centroid vector per category by averaging its example embeddings&#13;<br \/>\n        self.centroids = {}&#13;<br \/>\n        for cat, examples in categories.items():&#13;<br \/>\n            vecs = self.vectorizer.transform(examples).toarray()&#13;<br \/>\n            self.centroids[cat] = vecs.mean(axis=0)&#13;<br \/>\n&#13;<br \/>\n    def route(self, query: str) -&gt; dict:&#13;<br \/>\n        query_vec = self.vectorizer.transform([query]).toarray()[0]&#13;<br \/>\n        scores = {&#13;<br \/>\n            cat: float(cosine_similarity([query_vec], [centroid])[0][0])&#13;<br \/>\n            for cat, centroid in self.centroids.items()&#13;<br \/>\n        }&#13;<br \/>\n        best_cat   = max(scores, key=scores.get)&#13;<br \/>\n        best_score = scores[best_cat]&#13;<br \/>\n&#13;<br \/>\n        if best_score  {result[&#8216;category&#8217;]} (confidence={result[&#8216;confidence&#8217;]:.3f})&#8221;)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\" style=\"font-size: 12px !important; line-height: 15px !important;\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<p>47<\/p>\n<p>48<\/p>\n<p>49<\/p>\n<p>50<\/p>\n<p>51<\/p>\n<p>52<\/p>\n<p>53<\/p>\n<p>54<\/p>\n<p>55<\/p>\n<p>56<\/p>\n<p>57<\/p>\n<p>58<\/p>\n<p>59<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-p\"># router.py<\/span><\/p>\n<p><span class=\"crayon-p\"># Prerequisites: pip install scikit-learn numpy<\/span><\/p>\n<p><span class=\"crayon-p\"># Run: python router.py<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">np<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">feature_extraction<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">text <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">TfidfVectorizer<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">metrics<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">pairwise <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">cosine_similarity<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">CATEGORIES<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;data&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;query the database&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;read a file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;write data to storage&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;run a SQL query&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;communication&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;send an email&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;post a slack message&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;notify the team&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send a message&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;scheduling&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;create a calendar event&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;book a meeting&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;schedule an appointment&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">SemanticRouter<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Routes a query to a tool category using similarity against category<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0centroid embeddings. Falls back to &#8216;general&#8217; when no category clears<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0the confidence threshold &#8212; never guesses on a low-confidence match.<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">__init__<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">categories<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">confidence_threshold<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.15<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">threshold<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">confidence_threshold<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">TfidfVectorizer<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">all_examples<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-e\">ex <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">exs <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">categories<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">values<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">ex <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">exs<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">all_examples<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># Build one centroid vector per category by averaging its example embeddings<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">centroids<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">cat<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">examples <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">categories<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">items<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">vecs<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">examples<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">toarray<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">centroids<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">cat<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">vecs<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">axis<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">route<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">query_vec<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">toarray<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">scores<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">cat<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">cosine_similarity<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">query_vec<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">centroid<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">cat<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">centroid <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">centroids<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">items<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">best_cat<\/span><span class=\"crayon-h\">\u00a0\u00a0 <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">max<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">scores<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">key<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">scores<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">get<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">best_score<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">scores<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">best_cat<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">best_score<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\"><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">threshold<\/span><span class=\"crayon-o\">:<\/span><\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;category&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;general&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;confidence&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">best_score<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;category&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">best_cat<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;confidence&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">best_score<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">__name__<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;__main__&#8221;<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">router<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">SemanticRouter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">CATEGORIES<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">test_queries<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;Can you post a message in the sales Slack channel?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;I need to run a query against our production database&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;Schedule a meeting with the design team for tomorrow&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;asdkj qpwoe zxcv nonsense&#8221;<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">q<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">test_queries<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">router<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">route<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">q<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;&#8216;{q}&#8217; -&gt; {result[&#8216;category&#8217;]} (confidence={result[&#8216;confidence&#8217;]:.3f})&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>How to run:<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b555e803925919\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\npip install scikit-learn numpy&#13;<br \/>\npython router.py<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-e\">pip <\/span><span class=\"crayon-e\">install <\/span><span class=\"crayon-v\">scikit<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-e\">learn <\/span><span class=\"crayon-e\">numpy<\/span><\/p>\n<p><span class=\"crayon-e\">python <\/span><span class=\"crayon-v\">router<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">py<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The fallback to \u201c<strong>general<\/strong>\u201d on the gibberish query matters as much as the correct routes do. A router that always picks something, even on a query it has no real signal for, is more dangerous than one that admits it doesn\u2019t know.<\/p>\n<h2>Planner-Based Tool Selection<\/h2>\n<p>Retrieval and routing both answer \u201c<strong>what\u2019s relevant to this single turn<\/strong>.\u201d Multi-step tasks need something different: a sequence of tool calls planned upfront, with each step scoped to only the tools it specifically needs. This is the architecture that avoids what\u2019s sometimes called the God Agent anti-pattern \u2014 a single agent holding 20 tools in context with no plan structure \u2014 where a failure anywhere corrupts the whole task.<\/p>\n<p><strong>The pattern<\/strong>: ask the model to output a structured plan first, an ordered list of subtasks, each tagged with the capability it requires, before any tool executes. Then retrieve tools per step, scoped to that step\u2019s tag.<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b5563543910286\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\n# planner.py&#13;<br \/>\n# Prerequisites: none beyond Python&#8217;s standard library (json)&#13;<br \/>\n# Run: python planner.py&#13;<br \/>\n&#13;<br \/>\nimport json&#13;<br \/>\nfrom dataclasses import dataclass&#13;<br \/>\n&#13;<br \/>\n@dataclass&#13;<br \/>\nclass PlanStep:&#13;<br \/>\n    step_number: int&#13;<br \/>\n    description: str&#13;<br \/>\n    required_capability: str&#13;<br \/>\n&#13;<br \/>\ndef parse_plan(raw_plan_json: str) -&gt; list[PlanStep]:&#13;<br \/>\n    &#8220;&#8221;&#8221;Parse a planner LLM&#8217;s JSON output into structured PlanStep objects.&#8221;&#8221;&#8221;&#13;<br \/>\n    data = json.loads(raw_plan_json)&#13;<br \/>\n    return [&#13;<br \/>\n        PlanStep(s[&#8220;step_number&#8221;], s[&#8220;description&#8221;], s[&#8220;required_capability&#8221;])&#13;<br \/>\n        for s in data[&#8220;steps&#8221;]&#13;<br \/>\n    ]&#13;<br \/>\n&#13;<br \/>\n# Capability -&gt; tool-name mapping. In production this feeds the retriever&#13;<br \/>\n# from the previous section, scoped to only the tools tagged for this capability.&#13;<br \/>\nCAPABILITY_TOOLS = {&#13;<br \/>\n    &#8220;search&#8221;:        [&#8220;search_web&#8221;, &#8220;query_database&#8221;],&#13;<br \/>\n    &#8220;file_io&#8221;:       [&#8220;read_file&#8221;, &#8220;write_file&#8221;],&#13;<br \/>\n    &#8220;communication&#8221;: [&#8220;send_email&#8221;, &#8220;send_slack_message&#8221;],&#13;<br \/>\n    &#8220;synthesis&#8221;:     [&#8220;summarize_document&#8221;],&#13;<br \/>\n}&#13;<br \/>\n&#13;<br \/>\ndef get_scoped_tools(step: PlanStep) -&gt; list[str]:&#13;<br \/>\n    &#8220;&#8221;&#8221;Return only the tools relevant to this step &#8212; not the full catalog.&#8221;&#8221;&#8221;&#13;<br \/>\n    return CAPABILITY_TOOLS.get(step.required_capability, [])&#13;<br \/>\n&#13;<br \/>\n&#13;<br \/>\nif __name__ == &#8220;__main__&#8221;:&#13;<br \/>\n    # This JSON would normally come from an LLM call asking it to decompose&#13;<br \/>\n    # the task into steps, each tagged with the capability it needs.&#13;<br \/>\n    mock_plan = json.dumps({&#13;<br \/>\n        &#8220;steps&#8221;: [&#13;<br \/>\n            {&#8220;step_number&#8221;: 1, &#8220;description&#8221;: &#8220;Search for the latest sales report file&#8221;, &#8220;required_capability&#8221;: &#8220;search&#8221;},&#13;<br \/>\n            {&#8220;step_number&#8221;: 2, &#8220;description&#8221;: &#8220;Read the contents of the report file&#8221;, &#8220;required_capability&#8221;: &#8220;file_io&#8221;},&#13;<br \/>\n            {&#8220;step_number&#8221;: 3, &#8220;description&#8221;: &#8220;Summarize the key findings&#8221;, &#8220;required_capability&#8221;: &#8220;synthesis&#8221;},&#13;<br \/>\n            {&#8220;step_number&#8221;: 4, &#8220;description&#8221;: &#8220;Email the summary to the sales lead&#8221;, &#8220;required_capability&#8221;: &#8220;communication&#8221;},&#13;<br \/>\n        ]&#13;<br \/>\n    })&#13;<br \/>\n&#13;<br \/>\n    plan = parse_plan(mock_plan)&#13;<br \/>\n    for step in plan:&#13;<br \/>\n        scoped = get_scoped_tools(step)&#13;<br \/>\n        print(f&#8221;Step {step.step_number}: {step.description}&#8221;)&#13;<br \/>\n        print(f&#8221;  Capability: {step.required_capability} -&gt; tools available: {scoped}&#8221;)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\" style=\"font-size: 12px !important; line-height: 15px !important;\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<p>47<\/p>\n<p>48<\/p>\n<p>49<\/p>\n<p>50<\/p>\n<p>51<\/p>\n<p>52<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-p\"># planner.py<\/span><\/p>\n<p><span class=\"crayon-p\"># Prerequisites: none beyond Python&#8217;s standard library (json)<\/span><\/p>\n<p><span class=\"crayon-p\"># Run: python planner.py<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">json<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-e\">dataclasses <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-i\">dataclass<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-sy\">@<\/span><span class=\"crayon-e\">dataclass<\/span><\/p>\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">PlanStep<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">step_number<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">int<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">description<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">str<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">required_capability<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">str<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">parse_plan<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">raw_plan_json<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">PlanStep<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Parse a planner LLM&#8217;s JSON output into structured PlanStep objects.&#8221;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">json<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">loads<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">raw_plan_json<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">PlanStep<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">s<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;step_number&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">s<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">s<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;required_capability&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">s<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">data<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;steps&#8221;<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Capability -&gt; tool-name mapping. In production this feeds the retriever<\/span><\/p>\n<p><span class=\"crayon-p\"># from the previous section, scoped to only the tools tagged for this capability.<\/span><\/p>\n<p><span class=\"crayon-v\">CAPABILITY_TOOLS<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;search&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;search_web&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;query_database&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;file_io&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;read_file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;write_file&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;communication&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;send_email&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_slack_message&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;synthesis&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;summarize_document&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">get_scoped_tools<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">step<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">PlanStep<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Return only the tools relevant to this step &#8212; not the full catalog.&#8221;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">CAPABILITY_TOOLS<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">get<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">step<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">required_capability<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">__name__<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;__main__&#8221;<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># This JSON would normally come from an LLM call asking it to decompose<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># the task into steps, each tagged with the capability it needs.<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">mock_plan<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">json<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">dumps<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">{<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;steps&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;step_number&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Search for the latest sales report file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;required_capability&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;search&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;step_number&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Read the contents of the report file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;required_capability&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;file_io&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;step_number&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Summarize the key findings&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;required_capability&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;synthesis&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;step_number&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Email the summary to the sales lead&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;required_capability&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;communication&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">plan<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">parse_plan<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">mock_plan<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">step <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">plan<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">scoped<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">get_scoped_tools<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">step<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;Step {step.step_number}: {step.description}&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0Capability: {step.required_capability} -&gt; tools available: {scoped}&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>How to run (no dependencies required):<\/p>\n<p>Each step in this example sees one or two tools, never the full set. That\u2019s the actual mechanism behind why planning helps: it\u2019s not that the model reasons better when it has a plan; it\u2019s that the plan lets you legitimately narrow the tool list per step, which is the same lever retrieval pulls, applied at a finer grain.<\/p>\n<h2>Fallback Logic<\/h2>\n<p>Retrieval and routing both fail sometimes, not because the architecture is wrong, but because real queries are ambiguous, underspecified, or genuinely outside the tool catalog\u2019s coverage. What you do when the top match\u2019s confidence is low determines whether your agent degrades gracefully or starts guessing.<\/p>\n<p>A three-tier fallback chain handles this without resorting to a try\/except that just crashes the conversation: resolve directly when confidence is high, retry with a reformulated query when it isn\u2019t, and escalate to an explicit clarification request rather than forcing a tool call when even the retry comes up short.<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b5578567694366\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\n# fallback.py&#13;<br \/>\n# Prerequisites: pip install scikit-learn numpy&#13;<br \/>\n# Run: python fallback.py&#13;<br \/>\n&#13;<br \/>\nimport numpy as np&#13;<br \/>\nfrom sklearn.feature_extraction.text import TfidfVectorizer&#13;<br \/>\nfrom sklearn.metrics.pairwise import cosine_similarity&#13;<br \/>\n&#13;<br \/>\nTOOL_CATALOG = [&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;search_web&#8221;, &#8220;description&#8221;: &#8220;Search the web for current information on any topic&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;get_weather&#8221;, &#8220;description&#8221;: &#8220;Get the current weather forecast for a city&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;send_email&#8221;, &#8220;description&#8221;: &#8220;Send an email to a recipient with subject and body&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;list_github_issues&#8221;, &#8220;description&#8221;: &#8220;List open issues and bugs in a GitHub repository&#8221;},&#13;<br \/>\n]&#13;<br \/>\n&#13;<br \/>\nclass RetrieverWithFallback:&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    Wraps retrieval with a three-tier fallback chain:&#13;<br \/>\n    1. High confidence  -&gt; use the top result directly&#13;<br \/>\n    2. Low confidence   -&gt; retry with a reformulated query&#13;<br \/>\n    3. Still low        -&gt; escalate to a clarification request, never guess&#13;<br \/>\n    &#8220;&#8221;&#8221;&#13;<br \/>\n    def __init__(self, tools, confidence_threshold: float = 0.12):&#13;<br \/>\n        self.tools = tools&#13;<br \/>\n        self.threshold = confidence_threshold&#13;<br \/>\n        self.vectorizer = TfidfVectorizer()&#13;<br \/>\n        descs = [f&#8221;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221; for t in tools]&#13;<br \/>\n        self.tool_vectors = self.vectorizer.fit_transform(descs)&#13;<br \/>\n&#13;<br \/>\n    def _raw_retrieve(self, query: str):&#13;<br \/>\n        query_vec = self.vectorizer.transform([query])&#13;<br \/>\n        sims = cosine_similarity(query_vec, self.tool_vectors)[0]&#13;<br \/>\n        top_idx = int(np.argmax(sims))&#13;<br \/>\n        return self.tools[top_idx], float(sims[top_idx])&#13;<br \/>\n&#13;<br \/>\n    def retrieve_with_fallback(self, query: str) -&gt; dict:&#13;<br \/>\n        tool, score = self._raw_retrieve(query)&#13;<br \/>\n        if score &gt;= self.threshold:&#13;<br \/>\n            return {&#8220;status&#8221;: &#8220;resolved&#8221;, &#8220;tool&#8221;: tool[&#8220;name&#8221;], &#8220;confidence&#8221;: score, &#8220;attempts&#8221;: 1}&#13;<br \/>\n&#13;<br \/>\n        # Reformulate by stripping filler words. In production, this step would&#13;<br \/>\n        # be an LLM call asking it to restate the query in terms of intent\/capability.&#13;<br \/>\n        reformulated = query.replace(&#8220;can you&#8221;, &#8220;&#8221;).replace(&#8220;please&#8221;, &#8220;&#8221;).replace(&#8220;?&#8221;, &#8220;&#8221;).strip()&#13;<br \/>\n        tool2, score2 = self._raw_retrieve(reformulated)&#13;<br \/>\n        if score2 &gt;= self.threshold:&#13;<br \/>\n            return {&#8220;status&#8221;: &#8220;resolved&#8221;, &#8220;tool&#8221;: tool2[&#8220;name&#8221;], &#8220;confidence&#8221;: score2, &#8220;attempts&#8221;: 2}&#13;<br \/>\n&#13;<br \/>\n        return {&#13;<br \/>\n            &#8220;status&#8221;: &#8220;escalated&#8221;, &#8220;tool&#8221;: None,&#13;<br \/>\n            &#8220;confidence&#8221;: max(score, score2), &#8220;attempts&#8221;: 2,&#13;<br \/>\n            &#8220;clarification_request&#8221;: (&#13;<br \/>\n                f&#8221;I&#8217;m not confident which tool fits &#8216;{query}&#8217;. &#8220;&#13;<br \/>\n                f&#8221;Could you clarify what you&#8217;d like me to do?&#8221;&#13;<br \/>\n            ),&#13;<br \/>\n        }&#13;<br \/>\n&#13;<br \/>\n&#13;<br \/>\nif __name__ == &#8220;__main__&#8221;:&#13;<br \/>\n    retriever = RetrieverWithFallback(TOOL_CATALOG)&#13;<br \/>\n&#13;<br \/>\n    for q in [&#8220;What&#8217;s the weather forecast in Lagos?&#8221;, &#8220;xyzzy plugh random nonsense&#8221;]:&#13;<br \/>\n        result = retriever.retrieve_with_fallback(q)&#13;<br \/>\n        print(f&#8221;Query: &#8216;{q}'&#8221;)&#13;<br \/>\n        print(f&#8221;  Status: {result[&#8216;status&#8217;]}&#8221;)&#13;<br \/>\n        if result[&#8220;status&#8221;] == &#8220;resolved&#8221;:&#13;<br \/>\n            print(f&#8221;  Tool: {result[&#8216;tool&#8217;]} (confidence={result[&#8216;confidence&#8217;]:.3f})&#8221;)&#13;<br \/>\n        else:&#13;<br \/>\n            print(f&#8221;  {result[&#8216;clarification_request&#8217;]}&#8221;)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\" style=\"font-size: 12px !important; line-height: 15px !important;\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<p>47<\/p>\n<p>48<\/p>\n<p>49<\/p>\n<p>50<\/p>\n<p>51<\/p>\n<p>52<\/p>\n<p>53<\/p>\n<p>54<\/p>\n<p>55<\/p>\n<p>56<\/p>\n<p>57<\/p>\n<p>58<\/p>\n<p>59<\/p>\n<p>60<\/p>\n<p>61<\/p>\n<p>62<\/p>\n<p>63<\/p>\n<p>64<\/p>\n<p>65<\/p>\n<p>66<\/p>\n<p>67<\/p>\n<p>68<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-p\"># fallback.py<\/span><\/p>\n<p><span class=\"crayon-p\"># Prerequisites: pip install scikit-learn numpy<\/span><\/p>\n<p><span class=\"crayon-p\"># Run: python fallback.py<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">np<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">feature_extraction<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">text <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">TfidfVectorizer<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">metrics<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">pairwise <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">cosine_similarity<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">TOOL_CATALOG<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;search_web&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Search the web for current information on any topic&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;get_weather&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Get the current weather forecast for a city&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_email&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Send an email to a recipient with subject and body&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;list_github_issues&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;List open issues and bugs in a GitHub repository&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">RetrieverWithFallback<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Wraps retrieval with a three-tier fallback chain:<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a01. High confidence\u00a0\u00a0-&gt; use the top result directly<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a02. Low confidence\u00a0\u00a0 -&gt; retry with a reformulated query<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a03. Still low\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0-&gt; escalate to a clarification request, never guess<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">__init__<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">confidence_threshold<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.12<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">tools<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">threshold<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">confidence_threshold<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">TfidfVectorizer<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">descs<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">t<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tool_vectors<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit_transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">descs<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">_raw_retrieve<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">query_vec<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">sims<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cosine_similarity<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">query_vec<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tool_vectors<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">top_idx<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">int<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">argmax<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">sims<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">top_idx<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">sims<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">top_idx<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">retrieve_with_fallback<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">tool<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">_raw_retrieve<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">threshold<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;status&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;resolved&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;tool&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tool<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;confidence&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;attempts&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># Reformulate by stripping filler words. In production, this step would<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># be an LLM call asking it to restate the query in terms of intent\/capability.<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">reformulated<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">replace<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;can you&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">replace<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;please&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">replace<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">strip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">tool2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score2<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">_raw_retrieve<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">reformulated<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score2<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">threshold<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;status&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;resolved&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;tool&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tool2<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;confidence&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;attempts&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;status&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;escalated&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;tool&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;confidence&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">max<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">score<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">score2<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;attempts&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;clarification_request&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;I&#8217;m not confident which tool fits &#8216;{query}&#8217;. &#8220;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;Could you clarify what you&#8217;d like me to do?&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">__name__<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;__main__&#8221;<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">retriever<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">RetrieverWithFallback<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">TOOL_CATALOG<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">q<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;What&#8217;s the weather forecast in Lagos?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;xyzzy plugh random nonsense&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">retriever<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">retrieve_with_fallback<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">q<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;Query: &#8216;{q}'&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0Status: {result[&#8216;status&#8217;]}&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">result<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;status&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;resolved&#8221;<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0Tool: {result[&#8216;tool&#8217;]} (confidence={result[&#8216;confidence&#8217;]:.3f})&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">else<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0{result[&#8216;clarification_request&#8217;]}&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>How to run:<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b557f304275641\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\npip install scikit-learn numpy&#13;<br \/>\npython fallback.py<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-e\">pip <\/span><span class=\"crayon-e\">install <\/span><span class=\"crayon-v\">scikit<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-e\">learn <\/span><span class=\"crayon-e\">numpy<\/span><\/p>\n<p><span class=\"crayon-e\">python <\/span><span class=\"crayon-v\">fallback<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">py<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The escalation path is the one most teams skip when they first build this, and it\u2019s the one that matters most in production. A confidently wrong tool call is worse than a system that asks, \u201c<strong>I\u2019m not sure, could you clarify?<\/strong>\u201d The second failure mode is recoverable in one turn. The first one usually isn\u2019t.<\/p>\n<h2>Benchmarking Your Tool Selection System<\/h2>\n<p>Everything above is a hypothesis until you measure it. The methodology is straightforward: build a labeled set of (query, correct tool) pairs, run your pipeline against it, and measure accuracy, token cost, and latency, comparing your filtered pipeline against the naive full-catalog baseline. <a href=\"https:\/\/arxiv.org\/pdf\/2508.07575\" target=\"_blank\">MCPToolBench++<\/a>, a large-scale benchmark built from over 4,000 real MCP servers across 40-plus categories, is the reference for how rigorously this should be structured at scale, but the core idea works at any size.<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b5589694356361\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\n# benchmark.py&#13;<br \/>\n# Prerequisites: pip install scikit-learn numpy&#13;<br \/>\n# Run: python benchmark.py&#13;<br \/>\n&#13;<br \/>\nimport time&#13;<br \/>\nimport numpy as np&#13;<br \/>\nfrom sklearn.feature_extraction.text import TfidfVectorizer&#13;<br \/>\nfrom sklearn.metrics.pairwise import cosine_similarity&#13;<br \/>\n&#13;<br \/>\nTOOL_CATALOG = [&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;search_web&#8221;, &#8220;description&#8221;: &#8220;Search the web for current information on any topic&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;read_file&#8221;, &#8220;description&#8221;: &#8220;Read the contents of a file given its path&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;write_file&#8221;, &#8220;description&#8221;: &#8220;Write or overwrite content to a file at a given path&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;send_email&#8221;, &#8220;description&#8221;: &#8220;Send an email to a recipient with subject and body&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;create_calendar_event&#8221;, &#8220;description&#8221;: &#8220;Create a new calendar event with a title and time&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;query_database&#8221;, &#8220;description&#8221;: &#8220;Run a SQL query against the company database&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;list_github_issues&#8221;, &#8220;description&#8221;: &#8220;List open issues and bugs in a GitHub repository&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;send_slack_message&#8221;, &#8220;description&#8221;: &#8220;Send a message to a Slack channel or user&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;get_weather&#8221;, &#8220;description&#8221;: &#8220;Get current weather conditions for a city&#8221;},&#13;<br \/>\n    {&#8220;name&#8221;: &#8220;book_flight&#8221;, &#8220;description&#8221;: &#8220;Search and book a flight between two cities&#8221;},&#13;<br \/>\n]&#13;<br \/>\n&#13;<br \/>\n# Labeled benchmark set: (query, expected_tool). Build yours from real&#13;<br \/>\n# logged queries once you have production traffic &#8212; this is a seed set.&#13;<br \/>\nBENCHMARK_SET = [&#13;<br \/>\n    (&#8220;What&#8217;s the weather in Abuja right now?&#8221;, &#8220;get_weather&#8221;),&#13;<br \/>\n    (&#8220;Send an email to the finance team&#8221;, &#8220;send_email&#8221;),&#13;<br \/>\n    (&#8220;List the open issues on our main repo&#8221;, &#8220;list_github_issues&#8221;),&#13;<br \/>\n    (&#8220;Book me a flight from Lagos to London&#8221;, &#8220;book_flight&#8221;),&#13;<br \/>\n    (&#8220;Query the database for last week&#8217;s signups&#8221;, &#8220;query_database&#8221;),&#13;<br \/>\n    (&#8220;Post an update in the team Slack channel&#8221;, &#8220;send_slack_message&#8221;),&#13;<br \/>\n    (&#8220;Search the web for the latest interest rates&#8221;, &#8220;search_web&#8221;),&#13;<br \/>\n    (&#8220;Read the contents of config.yaml&#8221;, &#8220;read_file&#8221;),&#13;<br \/>\n]&#13;<br \/>\n&#13;<br \/>\ndef estimate_tokens(text: str) -&gt; int:&#13;<br \/>\n    &#8220;&#8221;&#8221;Rough token estimate (1 token ~ 4 characters) &#8212; good enough for relative comparison.&#8221;&#8221;&#8221;&#13;<br \/>\n    return len(text) \/\/ 4&#13;<br \/>\n&#13;<br \/>\nclass BenchmarkHarness:&#13;<br \/>\n    &#8220;&#8221;&#8221;Runs a labeled query set through a retriever and reports accuracy, token cost, and latency.&#8221;&#8221;&#8221;&#13;<br \/>\n&#13;<br \/>\n    def __init__(self, tools: list[dict], top_k: int = 3):&#13;<br \/>\n        self.tools = tools&#13;<br \/>\n        self.top_k = top_k&#13;<br \/>\n        self.vectorizer = TfidfVectorizer()&#13;<br \/>\n        descs = [f&#8221;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221; for t in tools]&#13;<br \/>\n        self.tool_vectors = self.vectorizer.fit_transform(descs)&#13;<br \/>\n        self.full_catalog_tokens = sum(estimate_tokens(d) for d in descs)&#13;<br \/>\n&#13;<br \/>\n    def _retrieve(self, query: str, top_k: int) -&gt; list[dict]:&#13;<br \/>\n        query_vec = self.vectorizer.transform([query])&#13;<br \/>\n        sims = cosine_similarity(query_vec, self.tool_vectors)[0]&#13;<br \/>\n        top_indices = np.argsort(sims)[::-1][:top_k]&#13;<br \/>\n        return [self.tools[i] for i in top_indices]&#13;<br \/>\n&#13;<br \/>\n    def run(self, benchmark_set: list[tuple], use_retrieval: bool = True) -&gt; dict:&#13;<br \/>\n        correct, total_tokens, latencies = 0, 0, []&#13;<br \/>\n&#13;<br \/>\n        for query, expected_tool in benchmark_set:&#13;<br \/>\n            t0 = time.perf_counter()&#13;<br \/>\n&#13;<br \/>\n            if use_retrieval:&#13;<br \/>\n                candidates = self._retrieve(query, top_k=self.top_k)&#13;<br \/>\n                tokens_this_query = sum(&#13;<br \/>\n                    estimate_tokens(f&#8221;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221;) for t in candidates&#13;<br \/>\n                )&#13;<br \/>\n            else:&#13;<br \/>\n                # Baseline: send the full, unfiltered catalog every time&#13;<br \/>\n                candidates = self.tools&#13;<br \/>\n                tokens_this_query = self.full_catalog_tokens&#13;<br \/>\n&#13;<br \/>\n            if expected_tool in [c[&#8220;name&#8221;] for c in candidates]:&#13;<br \/>\n                correct += 1&#13;<br \/>\n            total_tokens += tokens_this_query&#13;<br \/>\n            latencies.append(time.perf_counter() &#8211; t0)&#13;<br \/>\n&#13;<br \/>\n        n = len(benchmark_set)&#13;<br \/>\n        latencies_sorted = sorted(latencies)&#13;<br \/>\n        return {&#13;<br \/>\n            &#8220;accuracy&#8221;:       round(correct \/ n, 4),&#13;<br \/>\n            &#8220;avg_tokens&#8221;:     round(total_tokens \/ n, 1),&#13;<br \/>\n            &#8220;p50_latency_ms&#8221;: round(latencies_sorted[len(latencies_sorted) \/\/ 2] * 1000, 3),&#13;<br \/>\n            &#8220;p95_latency_ms&#8221;: round(latencies_sorted[int(len(latencies_sorted) * 0.95)] * 1000, 3),&#13;<br \/>\n        }&#13;<br \/>\n&#13;<br \/>\n&#13;<br \/>\nif __name__ == &#8220;__main__&#8221;:&#13;<br \/>\n    harness = BenchmarkHarness(TOOL_CATALOG, top_k=3)&#13;<br \/>\n&#13;<br \/>\n    baseline  = harness.run(BENCHMARK_SET, use_retrieval=False)&#13;<br \/>\n    retrieval = harness.run(BENCHMARK_SET, use_retrieval=True)&#13;<br \/>\n&#13;<br \/>\n    print(&#8220;Baseline (full catalog every time):&#8221;)&#13;<br \/>\n    print(f&#8221;  Accuracy:         {baseline[&#8216;accuracy&#8217;]*100:.1f}%&#8221;)&#13;<br \/>\n    print(f&#8221;  Avg tokens\/query: {baseline[&#8216;avg_tokens&#8217;]}&#8221;)&#13;<br \/>\n&#13;<br \/>\n    print(&#8220;\\nRetrieval-filtered (top-3):&#8221;)&#13;<br \/>\n    print(f&#8221;  Accuracy:         {retrieval[&#8216;accuracy&#8217;]*100:.1f}%&#8221;)&#13;<br \/>\n    print(f&#8221;  Avg tokens\/query: {retrieval[&#8216;avg_tokens&#8217;]}&#8221;)&#13;<br \/>\n&#13;<br \/>\n    reduction = (1 &#8211; retrieval[&#8220;avg_tokens&#8221;] \/ baseline[&#8220;avg_tokens&#8221;]) * 100&#13;<br \/>\n    print(f&#8221;\\nToken reduction with retrieval: {reduction:.1f}%&#8221;)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\" style=\"font-size: 12px !important; line-height: 15px !important;\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<p>47<\/p>\n<p>48<\/p>\n<p>49<\/p>\n<p>50<\/p>\n<p>51<\/p>\n<p>52<\/p>\n<p>53<\/p>\n<p>54<\/p>\n<p>55<\/p>\n<p>56<\/p>\n<p>57<\/p>\n<p>58<\/p>\n<p>59<\/p>\n<p>60<\/p>\n<p>61<\/p>\n<p>62<\/p>\n<p>63<\/p>\n<p>64<\/p>\n<p>65<\/p>\n<p>66<\/p>\n<p>67<\/p>\n<p>68<\/p>\n<p>69<\/p>\n<p>70<\/p>\n<p>71<\/p>\n<p>72<\/p>\n<p>73<\/p>\n<p>74<\/p>\n<p>75<\/p>\n<p>76<\/p>\n<p>77<\/p>\n<p>78<\/p>\n<p>79<\/p>\n<p>80<\/p>\n<p>81<\/p>\n<p>82<\/p>\n<p>83<\/p>\n<p>84<\/p>\n<p>85<\/p>\n<p>86<\/p>\n<p>87<\/p>\n<p>88<\/p>\n<p>89<\/p>\n<p>90<\/p>\n<p>91<\/p>\n<p>92<\/p>\n<p>93<\/p>\n<p>94<\/p>\n<p>95<\/p>\n<p>96<\/p>\n<p>97<\/p>\n<p>98<\/p>\n<p>99<\/p>\n<p>100<\/p>\n<p>101<\/p>\n<p>102<\/p>\n<p>103<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-p\"># benchmark.py<\/span><\/p>\n<p><span class=\"crayon-p\"># Prerequisites: pip install scikit-learn numpy<\/span><\/p>\n<p><span class=\"crayon-p\"># Run: python benchmark.py<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">time<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">np<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">feature_extraction<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">text <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">TfidfVectorizer<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">metrics<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">pairwise <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">cosine_similarity<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-v\">TOOL_CATALOG<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;search_web&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Search the web for current information on any topic&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;read_file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Read the contents of a file given its path&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;write_file&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Write or overwrite content to a file at a given path&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_email&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Send an email to a recipient with subject and body&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;create_calendar_event&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Create a new calendar event with a title and time&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;query_database&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Run a SQL query against the company database&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;list_github_issues&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;List open issues and bugs in a GitHub repository&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_slack_message&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Send a message to a Slack channel or user&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;get_weather&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Get current weather conditions for a city&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">{<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;book_flight&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;description&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Search and book a flight between two cities&#8221;<\/span><span class=\"crayon-sy\">}<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Labeled benchmark set: (query, expected_tool). Build yours from real<\/span><\/p>\n<p><span class=\"crayon-p\"># logged queries once you have production traffic &#8212; this is a seed set.<\/span><\/p>\n<p><span class=\"crayon-v\">BENCHMARK_SET<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;What&#8217;s the weather in Abuja right now?&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;get_weather&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Send an email to the finance team&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_email&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;List the open issues on our main repo&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;list_github_issues&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Book me a flight from Lagos to London&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;book_flight&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Query the database for last week&#8217;s signups&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;query_database&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Post an update in the team Slack channel&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;send_slack_message&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Search the web for the latest interest rates&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;search_web&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Read the contents of config.yaml&#8221;<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;read_file&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">estimate_tokens<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">text<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">int<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Rough token estimate (1 token ~ 4 characters) &#8212; good enough for relative comparison.&#8221;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">text<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-c\">\/\/ 4<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">BenchmarkHarness<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Runs a labeled query set through a retriever and reports accuracy, token cost, and latency.&#8221;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">__init__<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">int<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">tools<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">top_k<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">TfidfVectorizer<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">descs<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">t<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tool_vectors<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">fit_transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">descs<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">full_catalog_tokens<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sum<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">estimate_tokens<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">d<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">d<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">descs<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">_retrieve<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">str<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">int<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">query_vec<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">vectorizer<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">transform<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">sims<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cosine_similarity<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">query_vec<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tool_vectors<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">top_indices<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">argsort<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">sims<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">::<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">tools<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">i<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">i<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_indices<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">run<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">benchmark_set<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">list<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">tuple<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">use_retrieval<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">bool<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">True<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dict<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">correct<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">total_tokens<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">latencies<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">expected_tool <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">benchmark_set<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">t0<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">time<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">perf_counter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">use_retrieval<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">candidates<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">_retrieve<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">query<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">tokens_this_query<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sum<\/span><span class=\"crayon-sy\">(<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">estimate_tokens<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;{t[&#8216;name&#8217;]}: {t[&#8216;description&#8217;]}&#8221;<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">t<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">candidates<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">else<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># Baseline: send the full, unfiltered catalog every time<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">candidates<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">tools<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">tokens_this_query<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">full_catalog_tokens<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">expected_tool <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">c<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;name&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">c<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">candidates<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">correct<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">total_tokens<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">tokens_this_query<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">latencies<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">append<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">time<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">perf_counter<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">t0<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">benchmark_set<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">latencies_sorted<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sorted<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">latencies<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">{<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;accuracy&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-e\">round<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">correct<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">\/<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;avg_tokens&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0 <\/span><span class=\"crayon-e\">round<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">total_tokens<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">\/<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;p50_latency_ms&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">round<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">latencies_sorted<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">latencies_sorted<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-c\">\/\/ 2] * 1000, 3),<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;p95_latency_ms&#8221;<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">round<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">latencies_sorted<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-t\">int<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">latencies_sorted<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.95<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1000<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">}<\/span><\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-st\">if<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">__name__<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;__main__&#8221;<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">harness<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">BenchmarkHarness<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">TOOL_CATALOG<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">top_k<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">baseline<\/span><span class=\"crayon-h\">\u00a0\u00a0<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">harness<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">run<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">BENCHMARK_SET<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">use_retrieval<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-t\">False<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">retrieval<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">harness<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">run<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">BENCHMARK_SET<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">use_retrieval<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-t\">True<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Baseline (full catalog every time):&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0Accuracy:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {baseline[&#8216;accuracy&#8217;]*100:.1f}%&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0Avg tokens\/query: {baseline[&#8216;avg_tokens&#8217;]}&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;\\nRetrieval-filtered (top-3):&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0Accuracy:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {retrieval[&#8216;accuracy&#8217;]*100:.1f}%&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8221;\u00a0\u00a0Avg tokens\/query: {retrieval[&#8216;avg_tokens&#8217;]}&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">reduction<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">retrieval<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;avg_tokens&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">\/<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">baseline<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-s\">&#8220;avg_tokens&#8221;<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">100<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-i\">f<\/span><span class=\"crayon-s\">&#8220;\\nToken reduction with retrieval: {reduction:.1f}%&#8221;<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>How to run:<\/p>\n<div id=\"urvanov-syntax-highlighter-6a4d31c8b55b1463465309\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-pc print-yes notranslate\" data-settings=\" touchscreen minimize scroll-mouseover disable-anim\" style=\" margin-top: 12px; margin-bottom: 12px; font-size: 12px !important; line-height: 15px !important;\">\n<p><textarea wrap=\"soft\" class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" style=\"-moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4; font-size: 12px !important; line-height: 15px !important;\"><br \/>\npip install scikit-learn numpy&#13;<br \/>\npython benchmark.py<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\" style=\"\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\" style=\"font-size: 12px !important; line-height: 15px !important; -moz-tab-size:4; -o-tab-size:4; -webkit-tab-size:4; tab-size:4;\">\n<p><span class=\"crayon-e\">pip <\/span><span class=\"crayon-e\">install <\/span><span class=\"crayon-v\">scikit<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-e\">learn <\/span><span class=\"crayon-e\">numpy<\/span><\/p>\n<p><span class=\"crayon-e\">python <\/span><span class=\"crayon-v\">benchmark<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">py<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>On this 10-tool catalog with an 8-query benchmark set, retrieval-filtering held accuracy steady while cutting average tokens per query by roughly 70%. The exact numbers will shift with your catalog and query set, but the comparison structure is what matters: you now have a repeatable way to answer \u201cdid this change actually help\u201d instead of relying on a handful of manual spot checks.<\/p>\n<h2>Wrapping up<\/h2>\n<p>These six techniques aren\u2019t competing options; they\u2019re layers. Gating filters out turns that need no tool at all, cheaply, before anything else runs. Retrieval or routing narrows the catalog down to what\u2019s actually relevant for the turns that remain. Planning sequences of multi-step tasks so each step only sees the tools it needs. Fallback logic catches the cases where the first attempt doesn\u2019t land cleanly. Benchmarking is how you know whether any of the above made a measurable difference, rather than just feeling better.<\/p>\n<p>The RAG-MCP result, with accuracy more than tripling and tokens cut by half, isn\u2019t an outlier. It\u2019s what happens predictably once you stop asking a model to read through a full phone book before every decision. None of these techniques requires a bigger model or a longer context window. They require treating the tool list itself as something to be designed, not just appended to.<\/p>\n<p><strong>Resources<\/strong>:<\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"<p>In this article, you will learn why agent accuracy degrades as a tool catalog grows, and six practical techniques for keeping tool selection accurate and efficient at scale. Topics we will cover include: Why adding more tools to an agent causes tool hallucination and accuracy loss, not just slower responses. How gating, retrieval, routing, and<\/p>\n","protected":false},"author":1,"featured_media":180533,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[42],"tags":[],"class_list":{"0":"post-180532","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-ai"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>The Complete Guide to Tool Selection in AI Agents - Ktromedia<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/ktromedia.com\/?p=180532\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Complete Guide to Tool Selection in AI Agents - Ktromedia\" \/>\n<meta property=\"og:description\" content=\"In this article, you will learn why agent accuracy degrades as a tool catalog grows, and six practical techniques for keeping tool selection accurate and efficient at scale. Topics we will cover include: Why adding more tools to an agent causes tool hallucination and accuracy loss, not just slower responses. How gating, retrieval, routing, and\" \/>\n<meta property=\"og:url\" content=\"http:\/\/ktromedia.com\/?p=180532\" \/>\n<meta property=\"og:site_name\" content=\"Ktromedia\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/KTROMedia\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-07-07T18:39:12+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1536\" \/>\n\t<meta property=\"og:image:height\" content=\"1024\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"KTRO TEAM\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"KTRO TEAM\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"30 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"http:\/\/ktromedia.com\/?p=180532#article\",\"isPartOf\":{\"@id\":\"http:\/\/ktromedia.com\/?p=180532\"},\"author\":{\"name\":\"KTRO TEAM\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b\"},\"headline\":\"The Complete Guide to Tool Selection in AI Agents\",\"datePublished\":\"2026-07-07T18:39:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\/\/ktromedia.com\/?p=180532\"},\"wordCount\":6033,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/ktromedia.com\/#organization\"},\"image\":{\"@id\":\"http:\/\/ktromedia.com\/?p=180532#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png\",\"articleSection\":[\"\u4eba\u5de5\u667a\u80fd\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"http:\/\/ktromedia.com\/?p=180532#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/ktromedia.com\/?p=180532\",\"url\":\"http:\/\/ktromedia.com\/?p=180532\",\"name\":\"The Complete Guide to Tool Selection in AI Agents - Ktromedia\",\"isPartOf\":{\"@id\":\"https:\/\/ktromedia.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"http:\/\/ktromedia.com\/?p=180532#primaryimage\"},\"image\":{\"@id\":\"http:\/\/ktromedia.com\/?p=180532#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png\",\"datePublished\":\"2026-07-07T18:39:12+00:00\",\"breadcrumb\":{\"@id\":\"http:\/\/ktromedia.com\/?p=180532#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/ktromedia.com\/?p=180532\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/ktromedia.com\/?p=180532#primaryimage\",\"url\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png\",\"contentUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png\",\"width\":1536,\"height\":1024},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/ktromedia.com\/?p=180532#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ktromedia.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The Complete Guide to Tool Selection in AI Agents\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ktromedia.com\/#website\",\"url\":\"https:\/\/ktromedia.com\/\",\"name\":\"Ktromedia\",\"description\":\"KTRO MEDIA Crypto News\",\"publisher\":{\"@id\":\"https:\/\/ktromedia.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ktromedia.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ktromedia.com\/#organization\",\"name\":\"Ktromedia\",\"url\":\"https:\/\/ktromedia.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png\",\"contentUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png\",\"width\":250,\"height\":250,\"caption\":\"Ktromedia\"},\"image\":{\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/KTROMedia\/\",\"https:\/\/www.linkedin.com\/company\/ktro-media\/\",\"https:\/\/t.me\/ktrogroup\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b\",\"name\":\"KTRO TEAM\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png\",\"contentUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png\",\"caption\":\"KTRO TEAM\"},\"description\":\"KTRO MEDIA \u662f\u4e00\u5bb6\u5168\u7403\u6027\u7684\u534e\u6587WEB3\u5a92\u4f53\u516c\u53f8\u3002\u6211\u4eec\u81f4\u529b\u4e8e\u4e3a\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u9886\u57df\u63d0\u4f9b\u6700\u65b0\u7684\u65b0\u95fb\u3001\u89c1\u89e3\u548c\u8d8b\u52bf\u5206\u6790\u3002\u6211\u4eec\u7684\u5b97\u65e8\u662f\u4e3a\u5168\u7403\u7528\u6237\u63d0\u4f9b\u9ad8\u8d28\u91cf\u3001\u5168\u9762\u7684\u8d44\u8baf\u670d\u52a1\uff0c\u8ba9\u4ed6\u4eec\u66f4\u597d\u5730\u4e86\u89e3\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6700\u65b0\u52a8\u6001\u3002\u6211\u4eec\u4e5f\u5e0c\u671b\u80fd\u5e2e\u5230\u66f4\u591a\u4f18\u79c0\u7684WEB3\u4ea7\u54c1\u627e\u5230\u66f4\u591a\u66f4\u597d\u7684\u8d44\u6e90\u597d\u8ba9\u8fd9\u9886\u57df\u53d8\u5f97\u66f4\u6210\u719f\u3002 \u6211\u4eec\u7684\u62a5\u9053\u8303\u56f4\u6db5\u76d6\u4e86\u533a\u5757\u94fe\u3001\u52a0\u5bc6\u8d27\u5e01\u3001\u667a\u80fd\u5408\u7ea6\u3001DeFi\u3001NFT \u548c Web3 \u751f\u6001\u7cfb\u7edf\u7b49\u9886\u57df\u3002\u6211\u4eec\u7684\u62a5\u9053\u4e0d\u4ec5\u6765\u81ea\u884c\u4e1a\u5185\u7684\u4e13\u5bb6\uff0c\u5148\u950b\u8005\u4e5f\u5305\u62ec\u4e86\u6211\u4eec\u81ea\u5df1\u7684\u5206\u6790\u548c\u89c2\u70b9\u3002\u6211\u4eec\u5728\u5404\u4e2a\u56fd\u5bb6\u548c\u5730\u533a\u90fd\u8bbe\u6709\u56e2\u961f\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u672c\u5730\u5316\u7684\u62a5\u9053\u548c\u5206\u6790\u3002 \u9664\u4e86\u65b0\u95fb\u62a5\u9053\uff0c\u6211\u4eec\u8fd8\u63d0\u4f9b\u5e02\u573a\u7814\u7a76\u548c\u54a8\u8be2\u670d\u52a1\u3002\u6211\u4eec\u7684\u4e13\u4e1a\u56e2\u961f\u53ef\u4ee5\u4e3a\u60a8\u63d0\u4f9b\u6709\u5173\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6df1\u5165\u5206\u6790\u548c\u5e02\u573a\u8d8b\u52bf\uff0c\u5e2e\u52a9\u60a8\u505a\u51fa\u66f4\u660e\u667a\u7684\u6295\u8d44\u51b3\u7b56\u3002 \u6211\u4eec\u7684\u4f7f\u547d\u662f\u6210\u4e3a\u5168\u7403\u534e\u6587\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u6700\u53d7\u4fe1\u8d56\u7684\u4fe1\u606f\u6765\u6e90\u4e4b\u4e00\u3002\u6211\u4eec\u5c06\u7ee7\u7eed\u4e0d\u65ad\u52aa\u529b\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u6700\u65b0\u3001\u6700\u5168\u9762\u3001\u6700\u53ef\u9760\u7684\u4fe1\u606f\u670d\u52a1\u3002\",\"sameAs\":[\"https:\/\/ktromedia.com\"],\"url\":\"https:\/\/ktromedia.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Complete Guide to Tool Selection in AI Agents - Ktromedia","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/ktromedia.com\/?p=180532","og_locale":"en_US","og_type":"article","og_title":"The Complete Guide to Tool Selection in AI Agents - Ktromedia","og_description":"In this article, you will learn why agent accuracy degrades as a tool catalog grows, and six practical techniques for keeping tool selection accurate and efficient at scale. Topics we will cover include: Why adding more tools to an agent causes tool hallucination and accuracy loss, not just slower responses. How gating, retrieval, routing, and","og_url":"http:\/\/ktromedia.com\/?p=180532","og_site_name":"Ktromedia","article_publisher":"https:\/\/www.facebook.com\/KTROMedia\/","article_published_time":"2026-07-07T18:39:12+00:00","og_image":[{"width":1536,"height":1024,"url":"http:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png","type":"image\/png"}],"author":"KTRO TEAM","twitter_card":"summary_large_image","twitter_misc":{"Written by":"KTRO TEAM","Est. reading time":"30 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/ktromedia.com\/?p=180532#article","isPartOf":{"@id":"http:\/\/ktromedia.com\/?p=180532"},"author":{"name":"KTRO TEAM","@id":"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b"},"headline":"The Complete Guide to Tool Selection in AI Agents","datePublished":"2026-07-07T18:39:12+00:00","mainEntityOfPage":{"@id":"http:\/\/ktromedia.com\/?p=180532"},"wordCount":6033,"commentCount":0,"publisher":{"@id":"https:\/\/ktromedia.com\/#organization"},"image":{"@id":"http:\/\/ktromedia.com\/?p=180532#primaryimage"},"thumbnailUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png","articleSection":["\u4eba\u5de5\u667a\u80fd"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["http:\/\/ktromedia.com\/?p=180532#respond"]}]},{"@type":"WebPage","@id":"http:\/\/ktromedia.com\/?p=180532","url":"http:\/\/ktromedia.com\/?p=180532","name":"The Complete Guide to Tool Selection in AI Agents - Ktromedia","isPartOf":{"@id":"https:\/\/ktromedia.com\/#website"},"primaryImageOfPage":{"@id":"http:\/\/ktromedia.com\/?p=180532#primaryimage"},"image":{"@id":"http:\/\/ktromedia.com\/?p=180532#primaryimage"},"thumbnailUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png","datePublished":"2026-07-07T18:39:12+00:00","breadcrumb":{"@id":"http:\/\/ktromedia.com\/?p=180532#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/ktromedia.com\/?p=180532"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/ktromedia.com\/?p=180532#primaryimage","url":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png","contentUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/07\/The-Complete-Guide-to-Tool-Selection-in-AI-Agents.png","width":1536,"height":1024},{"@type":"BreadcrumbList","@id":"http:\/\/ktromedia.com\/?p=180532#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ktromedia.com\/"},{"@type":"ListItem","position":2,"name":"The Complete Guide to Tool Selection in AI Agents"}]},{"@type":"WebSite","@id":"https:\/\/ktromedia.com\/#website","url":"https:\/\/ktromedia.com\/","name":"Ktromedia","description":"KTRO MEDIA Crypto News","publisher":{"@id":"https:\/\/ktromedia.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ktromedia.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ktromedia.com\/#organization","name":"Ktromedia","url":"https:\/\/ktromedia.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/","url":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png","contentUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png","width":250,"height":250,"caption":"Ktromedia"},"image":{"@id":"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/KTROMedia\/","https:\/\/www.linkedin.com\/company\/ktro-media\/","https:\/\/t.me\/ktrogroup"]},{"@type":"Person","@id":"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b","name":"KTRO TEAM","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ktromedia.com\/#\/schema\/person\/image\/","url":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png","contentUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png","caption":"KTRO TEAM"},"description":"KTRO MEDIA \u662f\u4e00\u5bb6\u5168\u7403\u6027\u7684\u534e\u6587WEB3\u5a92\u4f53\u516c\u53f8\u3002\u6211\u4eec\u81f4\u529b\u4e8e\u4e3a\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u9886\u57df\u63d0\u4f9b\u6700\u65b0\u7684\u65b0\u95fb\u3001\u89c1\u89e3\u548c\u8d8b\u52bf\u5206\u6790\u3002\u6211\u4eec\u7684\u5b97\u65e8\u662f\u4e3a\u5168\u7403\u7528\u6237\u63d0\u4f9b\u9ad8\u8d28\u91cf\u3001\u5168\u9762\u7684\u8d44\u8baf\u670d\u52a1\uff0c\u8ba9\u4ed6\u4eec\u66f4\u597d\u5730\u4e86\u89e3\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6700\u65b0\u52a8\u6001\u3002\u6211\u4eec\u4e5f\u5e0c\u671b\u80fd\u5e2e\u5230\u66f4\u591a\u4f18\u79c0\u7684WEB3\u4ea7\u54c1\u627e\u5230\u66f4\u591a\u66f4\u597d\u7684\u8d44\u6e90\u597d\u8ba9\u8fd9\u9886\u57df\u53d8\u5f97\u66f4\u6210\u719f\u3002 \u6211\u4eec\u7684\u62a5\u9053\u8303\u56f4\u6db5\u76d6\u4e86\u533a\u5757\u94fe\u3001\u52a0\u5bc6\u8d27\u5e01\u3001\u667a\u80fd\u5408\u7ea6\u3001DeFi\u3001NFT \u548c Web3 \u751f\u6001\u7cfb\u7edf\u7b49\u9886\u57df\u3002\u6211\u4eec\u7684\u62a5\u9053\u4e0d\u4ec5\u6765\u81ea\u884c\u4e1a\u5185\u7684\u4e13\u5bb6\uff0c\u5148\u950b\u8005\u4e5f\u5305\u62ec\u4e86\u6211\u4eec\u81ea\u5df1\u7684\u5206\u6790\u548c\u89c2\u70b9\u3002\u6211\u4eec\u5728\u5404\u4e2a\u56fd\u5bb6\u548c\u5730\u533a\u90fd\u8bbe\u6709\u56e2\u961f\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u672c\u5730\u5316\u7684\u62a5\u9053\u548c\u5206\u6790\u3002 \u9664\u4e86\u65b0\u95fb\u62a5\u9053\uff0c\u6211\u4eec\u8fd8\u63d0\u4f9b\u5e02\u573a\u7814\u7a76\u548c\u54a8\u8be2\u670d\u52a1\u3002\u6211\u4eec\u7684\u4e13\u4e1a\u56e2\u961f\u53ef\u4ee5\u4e3a\u60a8\u63d0\u4f9b\u6709\u5173\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6df1\u5165\u5206\u6790\u548c\u5e02\u573a\u8d8b\u52bf\uff0c\u5e2e\u52a9\u60a8\u505a\u51fa\u66f4\u660e\u667a\u7684\u6295\u8d44\u51b3\u7b56\u3002 \u6211\u4eec\u7684\u4f7f\u547d\u662f\u6210\u4e3a\u5168\u7403\u534e\u6587\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u6700\u53d7\u4fe1\u8d56\u7684\u4fe1\u606f\u6765\u6e90\u4e4b\u4e00\u3002\u6211\u4eec\u5c06\u7ee7\u7eed\u4e0d\u65ad\u52aa\u529b\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u6700\u65b0\u3001\u6700\u5168\u9762\u3001\u6700\u53ef\u9760\u7684\u4fe1\u606f\u670d\u52a1\u3002","sameAs":["https:\/\/ktromedia.com"],"url":"https:\/\/ktromedia.com\/?author=1"}]}},"_links":{"self":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts\/180532","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=180532"}],"version-history":[{"count":1,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts\/180532\/revisions"}],"predecessor-version":[{"id":180534,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts\/180532\/revisions\/180534"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/media\/180533"}],"wp:attachment":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=180532"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=180532"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=180532"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}