From ed45b885911874396d42eee5d62c38c39d98ad49 Mon Sep 17 00:00:00 2001 From: Arjun Natarajan Date: Wed, 11 Jun 2025 13:58:38 -0400 Subject: [PATCH] docs: studio evals --- docs/docs/cloud/how-tos/studio/run_evals.md | 37 +++++++++++++++++++++ docs/docs/concepts/langgraph_studio.md | 3 +- docs/mkdocs.yml | 1 + 3 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 docs/docs/cloud/how-tos/studio/run_evals.md diff --git a/docs/docs/cloud/how-tos/studio/run_evals.md b/docs/docs/cloud/how-tos/studio/run_evals.md new file mode 100644 index 000000000..3b4864d27 --- /dev/null +++ b/docs/docs/cloud/how-tos/studio/run_evals.md @@ -0,0 +1,37 @@ +# Run experiments over a dataset + +LangGraph Studio supports evaluations by allowing you to run your assistant over a pre-defined LangSmith dataset. This enables you to understand how your application performs over a variety of inputs, compare the results to reference outputs, and score the results using evaluators. + +This guide shows you how to run an experiment in Studio. + +!!! info "Note" + + This feature is supported for all applications deployed on LangGraph Platform, or locally running Python applications. Support for the local JS server is coming soon. + +1. To start, ensure you have an existing LangSmith dataset. For information on how to create a dataset, see [here](https://docs.smith.langchain.com/evaluation/how_to_guides/manage_datasets_in_application#set-up-your-dataset). + +2. You can additionally choose to attach evaluators to the created dataset. Evaluators are functions that score how well your application performs on a given example. They get triggered at the end of the experiment and run against the experiment results. For more information on evaluators, see [here](https://docs.smith.langchain.com/evaluation/concepts#evaluators). + +3. Click "Run experiment" in the top right corner of the Studio page. + +4. Select the desired dataset (or dataset split) and click start. + +5. All of the inputs in the dataset will now be run against the active graph and assistant. + +6. Monitor the experiment progress in the top right corner. Click the arrow icon button at any time to view the experiment results. + +## Troubleshooting + +### Run experiment button disabled + +#### Deployed application + +If your application is deployed on LangGraph Platform, make a new revision to enable this feature. + +#### Local development server + +If you are running your application using the local server, make sure you upgrade to the latest version of the `langgraph-cli`. Additionally, you have tracing enabled via the `LANGSMITH_API_KEY` in your project's `.env` file. For local servers, this is not yet supported in JavaScript. + +### Evaluator results missing + +When you run your experiment, the evaluators get scheduled for execution in a queue. If you don't see evaluator results for the experiment, it likely means that these are still in the queue. Continue to check the experiment for updated results. diff --git a/docs/docs/concepts/langgraph_studio.md b/docs/docs/concepts/langgraph_studio.md index cdd8ef00c..6011de041 100644 --- a/docs/docs/concepts/langgraph_studio.md +++ b/docs/docs/concepts/langgraph_studio.md @@ -24,6 +24,7 @@ Key features of LangGraph Studio: - [Manage assistants](../cloud/how-tos/studio/manage_assistants.md) - [Manage threads](../cloud/how-tos/threads_studio.md) - [Iterate on prompts](../cloud/how-tos/iterate_graph_studio.md) +- [Run experiments over a dataset](../cloud/how-tos/studio/run_evals.md) - Manage [long term memory](memory.md) - Debug agent state via [time travel](time-travel.md) @@ -41,4 +42,4 @@ Chat mode is a simpler UI for iterating on and testing chat-specific agents. It ## Learn more -- See this guide on how to [get started](../cloud/how-tos/studio/quick_start.md) with LangGraph Studio. \ No newline at end of file +- See this guide on how to [get started](../cloud/how-tos/studio/quick_start.md) with LangGraph Studio. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 5c8ff289d..ed33e0b75 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -179,6 +179,7 @@ nav: - cloud/how-tos/studio/manage_assistants.md - cloud/how-tos/threads_studio.md - cloud/how-tos/iterate_graph_studio.md + - cloud/how-tos/studio/run_evals.md - cloud/how-tos/clone_traces_studio.md - cloud/how-tos/datasets_studio.md - LangGraph SDK: concepts/sdk.md