diff --git a/docs/docs/agents/models.md b/docs/docs/agents/models.md index b7929c096..6b8af56a7 100644 --- a/docs/docs/agents/models.md +++ b/docs/docs/agents/models.md @@ -1,233 +1,78 @@ ---- -search: - boost: 2 -tags: - - anthropic - - openai - - agent -hide: - - tags ---- - # Models -This page describes how to configure the chat model used by an agent. - -## Tool calling support - -To enable tool-calling agents, the underlying LLM must support [tool calling](https://python.langchain.com/docs/concepts/tool_calling/). - -Compatible models can be found in the [LangChain integrations directory](https://python.langchain.com/docs/integrations/chat/). - -## Specifying a model by name - -You can configure an agent with a model name string: - -=== "OpenAI" - - ```python - import os - from langgraph.prebuilt import create_react_agent - - os.environ["OPENAI_API_KEY"] = "sk-..." - - agent = create_react_agent( - # highlight-next-line - model="openai:gpt-4.1", - # other parameters - ) - ``` - -=== "Anthropic" - - ```python - import os - from langgraph.prebuilt import create_react_agent - - os.environ["ANTHROPIC_API_KEY"] = "sk-..." - - agent = create_react_agent( - # highlight-next-line - model="anthropic:claude-3-7-sonnet-latest", - # other parameters - ) - ``` - -=== "Azure" - - ```python - import os - from langgraph.prebuilt import create_react_agent - - os.environ["AZURE_OPENAI_API_KEY"] = "..." - os.environ["AZURE_OPENAI_ENDPOINT"] = "..." - os.environ["OPENAI_API_VERSION"] = "2025-03-01-preview" - - agent = create_react_agent( - # highlight-next-line - model="azure_openai:gpt-4.1", - # other parameters - ) - ``` - -=== "Google Gemini" - - ```python - import os - from langgraph.prebuilt import create_react_agent - - os.environ["GOOGLE_API_KEY"] = "..." - - agent = create_react_agent( - # highlight-next-line - model="google_genai:gemini-2.0-flash", - # other parameters - ) - ``` - -=== "AWS Bedrock" - - ```python - from langgraph.prebuilt import create_react_agent - - # Follow the steps here to configure your credentials: - # https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html - - agent = create_react_agent( - # highlight-next-line - model="bedrock_converse:anthropic.claude-3-5-sonnet-20240620-v1:0", - # other parameters - ) - ``` +LangGraph provides built-in support for [LLMs (language models)](https://python.langchain.com/docs/concepts/chat_models/) via the LangChain library. This makes it easy to integrate various LLMs into your agents and workflows. -## Using `init_chat_model` +## Initialize a model -The [`init_chat_model`](https://python.langchain.com/docs/how_to/chat_models_universal_init/) utility simplifies model initialization with configurable parameters: +Use [`init_chat_model`](https://python.langchain.com/docs/how_to/chat_models_universal_init/) to initialize models: -=== "OpenAI" +{!snippets/chat_model_tabs.md!} - ``` - pip install -U "langchain[openai]" - ``` - ```python - import os - from langchain.chat_models import init_chat_model - - os.environ["OPENAI_API_KEY"] = "sk-..." - - model = init_chat_model( - "openai:gpt-4.1", - temperature=0, - # other parameters - ) - ``` - -=== "Anthropic" - - ``` - pip install -U "langchain[anthropic]" - ``` - ```python - import os - from langchain.chat_models import init_chat_model - - os.environ["ANTHROPIC_API_KEY"] = "sk-..." - - model = init_chat_model( - "anthropic:claude-3-5-sonnet-latest", - temperature=0, - # other parameters - ) - ``` - -=== "Azure" - - ``` - pip install -U "langchain[openai]" - ``` - ```python - import os - from langchain.chat_models import init_chat_model - - os.environ["AZURE_OPENAI_API_KEY"] = "..." - os.environ["AZURE_OPENAI_ENDPOINT"] = "..." - os.environ["OPENAI_API_VERSION"] = "2025-03-01-preview" - - model = init_chat_model( - "azure_openai:gpt-4.1", - azure_deployment=os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"], - temperature=0, - # other parameters - ) - ``` - -=== "Google Gemini" - - ``` - pip install -U "langchain[google-genai]" - ``` - ```python - import os - from langchain.chat_models import init_chat_model - - os.environ["GOOGLE_API_KEY"] = "..." - - model = init_chat_model( - "google_genai:gemini-2.0-flash", - temperature=0, - # other parameters - ) - ``` - -=== "AWS Bedrock" - - ``` - pip install -U "langchain[aws]" - ``` - ```python - from langchain.chat_models import init_chat_model - - # Follow the steps here to configure your credentials: - # https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html - - model = init_chat_model( - "anthropic.claude-3-5-sonnet-20240620-v1:0", - model_provider="bedrock_converse", - temperature=0, - # other parameters - ) - ``` - - -Refer to the [API reference](https://python.langchain.com/api_reference/langchain/chat_models/langchain.chat_models.base.init_chat_model.html) for advanced options. - -## Using provider-specific LLMs +### Instantiate a model directly If a model provider is not available via `init_chat_model`, you can instantiate the provider's model class directly. The model must implement the [BaseChatModel interface](https://python.langchain.com/api_reference/core/language_models/langchain_core.language_models.chat_models.BaseChatModel.html) and support tool calling: + ```python +# Anthropic is already supported by `init_chat_model`, +# but you can also instantiate it directly. from langchain_anthropic import ChatAnthropic -from langgraph.prebuilt import create_react_agent model = ChatAnthropic( - model="claude-3-7-sonnet-latest", - temperature=0, - max_tokens=2048 -) - -agent = create_react_agent( - # highlight-next-line - model=model, - # other parameters + model="claude-3-7-sonnet-latest", + temperature=0, + max_tokens=2048 ) ``` -!!! note "Illustrative example" +!!! important "Tool calling support" - The example above uses `ChatAnthropic`, which is already supported by `init_chat_model`. This pattern is shown to illustrate how to manually instantiate a model not available through init_chat_model. + If you are building an agent or workflow that requires the model to call external tools, ensure that the underlying + language model supports [tool calling](../concepts/tools.md). Compatible models can be found in the [LangChain integrations directory](https://python.langchain.com/docs/integrations/chat/). -## Disable streaming + +## Use in an agent + +When using `create_react_agent` you can specify the model by its name string, which is a shorthand for initializing the model using `init_chat_model`. This allows you to use the model without needing to import or instantiate it directly. + +=== "model name" + + + ```python + from langgraph.prebuilt import create_react_agent + + create_react_agent( + # highlight-next-line + model="anthropic:claude-3-7-sonnet-latest", + # other parameters + ) + ``` + +=== "model instance" + + ```python + from langchain_anthropic import ChatAnthropic + from langgraph.prebuilt import create_react_agent + + model = ChatAnthropic( + model="claude-3-7-sonnet-latest", + temperature=0, + max_tokens=2048 + ) + # Alternatively + # model = init_chat_model("anthropic:claude-3-7-sonnet-latest") + + agent = create_react_agent( + # highlight-next-line + model=model, + # other parameters + ) + ``` + +## Advanced model configuration + +### Disable streaming To disable streaming of the individual LLM tokens, set `disable_streaming=True` when initializing the model: @@ -257,7 +102,7 @@ To disable streaming of the individual LLM tokens, set `disable_streaming=True` Refer to the [API reference](https://python.langchain.com/api_reference/core/language_models/langchain_core.language_models.chat_models.BaseChatModel.html#langchain_core.language_models.chat_models.BaseChatModel.disable_streaming) for more information on `disable_streaming` -## Adding model fallbacks +### Add model fallbacks You can add a fallback to a different model or a different LLM provider using `model.with_fallbacks([...])`: @@ -292,7 +137,43 @@ You can add a fallback to a different model or a different LLM provider using `m See this [guide](https://python.langchain.com/docs/how_to/fallbacks/#fallback-to-better-model) for more information on model fallbacks. +### Use the built-in rate limiter + +Langchain includes a built-in in-memory rate limiter. This rate limiter is thread safe and can be shared by multiple threads in the same process. + +```python +from langchain_core.rate_limiters import InMemoryRateLimiter +from langchain_anthropic import ChatAnthropic + +rate_limiter = InMemoryRateLimiter( + requests_per_second=0.1, # <-- Super slow! We can only make a request once every 10 seconds!! + check_every_n_seconds=0.1, # Wake up every 100 ms to check whether allowed to make a request, + max_bucket_size=10, # Controls the maximum burst size. +) + +model = ChatAnthropic( + model_name="claude-3-opus-20240229", + rate_limiter=rate_limiter +) +``` + +See the LangChain docs for more information on how to [handle rate limiting](https://python.langchain.com/docs/how_to/chat_model_rate_limiting/). + +## Bring your own model + +If your desired LLM isn't officially supported by LangChain, consider these options: + +1. **Implement a custom LangChain chat model**: Create a model conforming to the [LangChain chat model interface](https://python.langchain.com/docs/how_to/custom_chat_model/). This enables full compatibility with LangGraph's agents and workflows but requires understanding of the LangChain framework. + +2. **Direct invocation with custom streaming**: Use your model directly by [adding custom streaming logic](../how-tos/streaming.md#use-with-any-llm) with `StreamWriter`. + Refer to the [custom streaming documentation](../how-tos/streaming.md#use-with-any-llm) for guidance. This approach suits custom workflows where prebuilt agent integration is not necessary. + + ## Additional resources +- [Multimodal inputs](https://python.langchain.com/docs/how_to/multimodal_inputs/) +- [Structured outputs](https://python.langchain.com/docs/how_to/structured_output/) - [Model integration directory](https://python.langchain.com/docs/integrations/chat/) -- [Universal initialization with `init_chat_model`](https://python.langchain.com/docs/how_to/chat_models_universal_init/) +- [Force model to call a specific tool](https://python.langchain.com/docs/how_to/tool_choice/) +- [All chat model how-to guides](https://python.langchain.com/docs/how_to/#chat-models) +- [Chat model integrations](https://python.langchain.com/docs/integrations/chat/) diff --git a/docs/snippets/chat_model_tabs.md b/docs/snippets/chat_model_tabs.md index df318899f..e984e1821 100644 --- a/docs/snippets/chat_model_tabs.md +++ b/docs/snippets/chat_model_tabs.md @@ -1,6 +1,6 @@ === "OpenAI" - ``` + ```shell pip install -U "langchain[openai]" ``` ```python @@ -12,9 +12,11 @@ llm = init_chat_model("openai:gpt-4.1") ``` + 👉 Read the [OpenAI integration docs](https://python.langchain.com/docs/integrations/chat/openai/) + === "Anthropic" - ``` + ```shell pip install -U "langchain[anthropic]" ``` ```python @@ -26,9 +28,11 @@ llm = init_chat_model("anthropic:claude-3-5-sonnet-latest") ``` + 👉 Read the [Anthropic integration docs](https://python.langchain.com/docs/integrations/chat/anthropic/) + === "Azure" - ``` + ```shell pip install -U "langchain[openai]" ``` ```python @@ -44,10 +48,12 @@ azure_deployment=os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"], ) ``` + + 👉 Read the [Azure integration docs](https://python.langchain.com/docs/integrations/chat/azure_chat_openai/) === "Google Gemini" - ``` + ```shell pip install -U "langchain[google-genai]" ``` ```python @@ -59,9 +65,11 @@ llm = init_chat_model("google_genai:gemini-2.0-flash") ``` + 👉 Read the [Google GenAI integration docs](https://python.langchain.com/docs/integrations/chat/google_generative_ai/) + === "AWS Bedrock" - ``` + ```shell pip install -U "langchain[aws]" ``` ```python @@ -75,3 +83,5 @@ model_provider="bedrock_converse", ) ``` + + 👉 Read the [AWS Bedrock integration docs](https://python.langchain.com/docs/integrations/chat/bedrock/)