How to handle large numbers of tools

The subset of available tools to call is generally at the discretion of the model (although many providers also enable the user to specify or constrain the choice of tool). As the number of available tools grows, you may want to limit the scope of the LLM's selection, to decrease token consumption and to help manage sources of error in LLM reasoning.

Here we will demonstrate how to dynamically adjust the tools available to a model. Bottom line up front: like RAG and similar methods, we prefix the model invocation by retrieving over available tools. Although we demonstrate one implementation that searches over tool descriptions, the details of the tool selection can be customized as needed.

Define the tools

Let's consider a toy example in which we have one tool for each company in the S&P 500 index. Each tool will fetch information, and is parameterized by a single integer representing the year.

We first construct a registry that associates a unique identifier with a schema for each tool. We will represent the tools using JSON schema, which can be bound directly to chat models supporting tool calling.

In [1]:

import re
import uuid

from langchain_core.tools import StructuredTool


def create_tool(company: str) -> dict:
    """Create schema for a placeholder tool."""
    formatted_company = re.sub(r"[^\w\s]", "", company).replace(" ", "_")

    def company_tool(year: int) -> str:
        return f"{company} had revenues of $100 in {year}."

    return StructuredTool.from_function(
        company_tool,
        name=formatted_company,
        description=f"Information about {company}",
    )


s_and_p_500_companies = [  # Abbreviated list for demonstration purposes
    "3M",
    "A.O. Smith",
    "Abbott",
    "Accenture",
    "Advanced Micro Devices",
    "Yum! Brands",
    "Zebra Technologies",
    "Zimmer Biomet",
    "Zoetis",
]

tool_registry = {
    str(uuid.uuid4()): create_tool(company) for company in s_and_p_500_companies
}

Define the graph

Tool selection

We will construct a node that retrieves a subset of available tools given the information in the state-- such as a recent user message. In general, the full scope of retrieval solutions are available for this step. As a simple solution, we index embeddings of tool descriptions in a vector store, and associate user queries to tools via semantic search.

In [2]:

from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore, VectorStore
from langchain_openai import OpenAIEmbeddings

tool_documents = [
    Document(page_content=tool.description, id=id)
    for id, tool in tool_registry.items()
]

vector_store = InMemoryVectorStore(embedding=OpenAIEmbeddings())
upsert_response = vector_store.upsert(tool_documents)
assert not upsert_response["failed"]

Incorporating with an agent

We will use a typical React agent graph (e.g., as used in the quickstart), with some modifications:

We add a selected_tools key to the state, which stores our selected subset of tools;
We set the entry point of the graph to be a select_tools node, which populates this element of the state;
We bind the selected subset of tools to the chat model within the agent node.

In [3]:

from typing import Annotated

from langchain_openai import ChatOpenAI
from typing_extensions import TypedDict

from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition


class State(TypedDict):
    messages: Annotated[list, add_messages]
    selected_tools: list[str]


graph_builder = StateGraph(State)

tools = list(tool_registry.values())
llm = ChatOpenAI()


def agent(state: State):
    selected_tools = [tool_registry[id] for id in state["selected_tools"]]
    llm_with_tools = llm.bind_tools(selected_tools)
    return {"messages": [llm_with_tools.invoke(state["messages"])]}


def select_tools(state: State):
    last_user_message = state["messages"][-1]
    query = last_user_message.content
    tool_documents = vector_store.similarity_search(query)
    return {"selected_tools": [document.id for document in tool_documents]}


graph_builder.add_node("agent", agent)
graph_builder.add_node("select_tools", select_tools)

tool_node = ToolNode(tools=tools)
graph_builder.add_node("tools", tool_node)

graph_builder.add_conditional_edges(
    "agent",
    tools_condition,
)
graph_builder.add_edge("tools", "agent")
graph_builder.add_edge("select_tools", "agent")
graph_builder.add_edge(START, "select_tools")
graph = graph_builder.compile()

In [4]:

from IPython.display import Image, display

try:
    display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
    # This requires some extra dependencies and is optional
    pass

In [5]:

user_input = "Can you give me some information about AMD in 2022?"

result = graph.invoke({"messages": [("user", user_input)]})

In [6]:

print(result["selected_tools"])

['3a42d402-4f6f-44ce-9ff4-d53d50f5f24a', '42aafb13-52e4-402b-80aa-db5be2e0e40f', '275bcddc-a918-4688-938f-9d2d6f232757', 'abc00b37-e0fa-470f-98d7-f055cc29a11f']

In [7]:

for message in result["messages"]:
    message.pretty_print()

================================[1m Human Message [0m=================================

Can you give me some information about AMD in 2022?
==================================[1m Ai Message [0m==================================
Tool Calls:
  Advanced_Micro_Devices (call_oXCnqxGmzQaP4OTEWvPghiKR)
 Call ID: call_oXCnqxGmzQaP4OTEWvPghiKR
  Args:
    year: 2022
=================================[1m Tool Message [0m=================================
Name: Advanced_Micro_Devices

Advanced Micro Devices had revenues of $100 in 2022.
==================================[1m Ai Message [0m==================================

In 2022, Advanced Micro Devices had revenues of $100.

Next steps

This guide provides a minimal implementation for dynamically selecting tools. There is a host of possible improvements and optimizations:

Repeating tool selection: To manage errors from incorrect tool selection, we could revisit the select_tools node. Options include:
- Modify select_tools to generate the vector store query using all messages in the state (e.g., with a chat model) and add an edge routing from tools to select_tools;
- Equip the agent with a reselect_tools tool, allowing it to re-select tools at its discretion.
Optimizing tool selection: In general, the full scope of retrieval solutions are available for tool selection. Additional options include:
- Group tools and retrieve over groups;
- Use a chat model to select tools or groups of tool.

24 KiB Raw Blame History