langgraph/examples/cloud_examples/human-in-the-loop_cloud.ipynb

How to have a human in the loop

With it's built in persistence layer, LangGraph API is perfect for human-in-the-loop workflows. Here we cover a few such examples:

1. Having a human in the loop to approve a tool call
2. Having a human in the loop to edit a tool call
3. Having a human in the loop to edit an old state and resume execution from there

In [1]:

from langgraph_sdk import get_client

In [2]:

client = get_client()

In [3]:

# There should always be a default assistant with no configuration
assistants = await client.assistants.search()
assistants = [a for a in assistants if not a['config']]
assistants

Out [3]:

[{'assistant_id': 'fe096781-5601-53d2-b2f6-0d3403f7e9ca',
  'graph_id': 'agent',
  'config': {},
  'created_at': '2024-05-18T00:19:39.688822+00:00',
  'updated_at': '2024-05-18T00:19:39.688822+00:00',
  'metadata': {'created_by': 'system'}}]

In [4]:

assistant = assistants[0]
assistant

Out [4]:

{'assistant_id': 'fe096781-5601-53d2-b2f6-0d3403f7e9ca',
 'graph_id': 'agent',
 'config': {},
 'created_at': '2024-05-18T00:19:39.688822+00:00',
 'updated_at': '2024-05-18T00:19:39.688822+00:00',
 'metadata': {'created_by': 'system'}}

Approve a tool call

In [5]:

thread = await client.threads.create()
thread

Out [5]:

{'thread_id': '54ed0901-6767-46c9-a5f9-b65c1c5fd89c',
 'created_at': '2024-05-18T22:46:16.724701+00:00',
 'updated_at': '2024-05-18T22:46:16.724701+00:00',
 'metadata': {}}

In [6]:

runs = await client.runs.list(thread['thread_id'])
runs

Out [6]:

[]

We now want to add a human-in-the-loop step before a tool is called. We can do this by adding interrupt_before=["action"], which tells us to interrupt before calling the action node. We can do this either when compiling the graph or when kicking off a run. Here we will do it when kicking of a run.

In [7]:

input = {"messages": [{"role": "human", "content": "what's the weather in sf"}]}
async for chunk in client.runs.stream(
    thread['thread_id'], assistant['assistant_id'], input=input, stream_mode="updates", interrupt_before=['action']
):
    print(f"Receiving new event of type: {chunk.event}...")
    print(chunk.data)
    print("\n\n")

Receiving new event of type: metadata...
{'run_id': '3b77ef83-687a-4840-8858-0371f91a92c3'}



Receiving new event of type: data...
{'agent': {'messages': [{'content': [{'id': 'toolu_01HwZqM1ptX6E15A5LAmyZTB', 'input': {'query': 'weather in san francisco'}, 'name': 'tavily_search_results_json', 'type': 'tool_use'}], 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'ai', 'name': None, 'id': 'run-e5d17791-4d37-4ad2-815f-a0c4cba62585', 'example': False, 'tool_calls': [{'name': 'tavily_search_results_json', 'args': {'query': 'weather in san francisco'}, 'id': 'toolu_01HwZqM1ptX6E15A5LAmyZTB'}], 'invalid_tool_calls': []}]}}



Receiving new event of type: end...
None

We can now kick off a new run on the same thread with None as the input in order to just continue the existing thread.

In [8]:

input = None
async for chunk in client.runs.stream(
    thread['thread_id'], assistant['assistant_id'], input=input, stream_mode="updates", interrupt_before=['action']
):
    print(f"Receiving new event of type: {chunk.event}...")
    print(chunk.data)
    print("\n\n")

Receiving new event of type: metadata...
{'run_id': 'a46f733d-cf5b-4ee3-9e07-08612468c8df'}



Receiving new event of type: data...
{'action': {'messages': [{'content': '[{"url": "https://www.weatherapi.com/", "content": "{\'location\': {\'name\': \'San Francisco\', \'region\': \'California\', \'country\': \'United States of America\', \'lat\': 37.78, \'lon\': -122.42, \'tz_id\': \'America/Los_Angeles\', \'localtime_epoch\': 1716072201, \'localtime\': \'2024-05-18 15:43\'}, \'current\': {\'last_updated_epoch\': 1716071400, \'last_updated\': \'2024-05-18 15:30\', \'temp_c\': 18.9, \'temp_f\': 66.0, \'is_day\': 1, \'condition\': {\'text\': \'Partly cloudy\', \'icon\': \'//cdn.weatherapi.com/weather/64x64/day/116.png\', \'code\': 1003}, \'wind_mph\': 18.6, \'wind_kph\': 29.9, \'wind_degree\': 280, \'wind_dir\': \'W\', \'pressure_mb\': 1015.0, \'pressure_in\': 29.96, \'precip_mm\': 0.0, \'precip_in\': 0.0, \'humidity\': 59, \'cloud\': 25, \'feelslike_c\': 18.9, \'feelslike_f\': 66.0, \'vis_km\': 16.0, \'vis_miles\': 9.0, \'uv\': 5.0, \'gust_mph\': 23.0, \'gust_kph\': 37.1}}"}]', 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'tool', 'name': 'tavily_search_results_json', 'id': '8be98ff3-6d61-41c5-8384-8db6b7abdbfb', 'tool_call_id': 'toolu_01HwZqM1ptX6E15A5LAmyZTB'}]}}



Receiving new event of type: data...
{'agent': {'messages': [{'content': "The weather in San Francisco is currently partly cloudy with a temperature of around 66°F (18.9°C). There are westerly winds of 18.6 mph (29.9 km/h) with gusts up to 23 mph (37.1 km/h). The humidity is 59% and visibility is good at 9 miles (16 km). UV levels are moderate at 5.0.\n\nIn summary, it's a nice partly cloudy spring day in San Francisco with comfortable temperatures and a moderate breeze. The weather conditions seem ideal for being outdoors and enjoying the city.", 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'ai', 'name': None, 'id': 'run-7a8a2ff8-d0d6-4200-b0a5-926f2b6a4798', 'example': False, 'tool_calls': [], 'invalid_tool_calls': []}]}}



Receiving new event of type: end...
None

Edit a tool call

What if we want to edit the tool call? We can also do that. Let's kick off another run, with the same interrupt_before=['action']

In [9]:

input = {"messages": [{"role": "human", "content": "what's the weather in la?"}]}
async for chunk in client.runs.stream(
    thread['thread_id'], assistant['assistant_id'], input=input, stream_mode="updates", interrupt_before=['action']
):
    print(f"Receiving new event of type: {chunk.event}...")
    print(chunk.data)
    print("\n\n")

Receiving new event of type: metadata...
{'run_id': 'c7c8e313-dad9-47d9-bd03-e112c94eff9e'}



Receiving new event of type: data...
{'agent': {'messages': [{'content': [{'id': 'toolu_01NGhKmeciaT7TfhBSwUT3mi', 'input': {'query': 'weather in los angeles'}, 'name': 'tavily_search_results_json', 'type': 'tool_use'}], 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'ai', 'name': None, 'id': 'run-3d417aa5-e9c1-4b76-90f8-597519c28af9', 'example': False, 'tool_calls': [{'name': 'tavily_search_results_json', 'args': {'query': 'weather in los angeles'}, 'id': 'toolu_01NGhKmeciaT7TfhBSwUT3mi'}], 'invalid_tool_calls': []}]}}



Receiving new event of type: end...
None

We can now inspect the state of the thread

In [10]:

thread_state = await client.threads.get_state(thread['thread_id'])

Let's get the last message of the thread - this is the one we want to update

In [11]:

last_message = thread_state['values']['messages'][-1]

In [12]:

last_message['content']

Out [12]:

[{'id': 'toolu_01NGhKmeciaT7TfhBSwUT3mi',
  'input': {'query': 'weather in los angeles'},
  'name': 'tavily_search_results_json',
  'type': 'tool_use'}]

Let's now modify the tool call to say Louisiana

In [13]:

last_message['tool_calls'] = [{
    'id': last_message['tool_calls'][0]['id'],
    'name': 'tavily_search_results_json',
    # We change the query to say temperature
    'args': {'query': 'weather in Louisiana'}
}]
# last_message['content'] = [{
#     'id': last_message['content'][0]['id'],
#     'name': 'tavily_search_results_json',
#     # We change the query to say temperature
#     'input': {'query': 'weather in Louisiana'},
#     'type': 'tool_use'
# }]

We can now update the state - we only need to pass in the last updated message because our graph will handle the update.

In [14]:

await client.threads.update_state(thread['thread_id'], values={"messages": [last_message]})

Out [14]:

{'configurable': {'thread_id': '54ed0901-6767-46c9-a5f9-b65c1c5fd89c',
  'thread_ts': '1ef15688-1dbd-68f5-8007-75dc0e110124'}}

Let's now check the state of the thread again, and in particular the final message

In [15]:

thread_state = await client.threads.get_state(thread['thread_id'])
thread_state['values']['messages'][-1]['tool_calls']

Out [15]:

[{'name': 'tavily_search_results_json',
  'args': {'query': 'weather in Louisiana'},
  'id': 'toolu_01NGhKmeciaT7TfhBSwUT3mi'}]

Great! We changed it. If we now resume execution (by kicking off a new run with null inputs on the same thread) it should use that new tool call.

In [16]:

input = None
async for chunk in client.runs.stream(
    thread['thread_id'], assistant['assistant_id'], input=input, stream_mode="updates", interrupt_before=['action']
):
    print(f"Receiving new event of type: {chunk.event}...")
    print(chunk.data)
    print("\n\n")

Receiving new event of type: metadata...
{'run_id': '1a1ebed1-3581-418a-81be-e834b40c5c82'}



Receiving new event of type: data...
{'action': {'messages': [{'content': '[{"url": "https://www.weatherapi.com/", "content": "{\'location\': {\'name\': \'Louisiana\', \'region\': \'Missouri\', \'country\': \'USA United States of America\', \'lat\': 39.44, \'lon\': -91.06, \'tz_id\': \'America/Chicago\', \'localtime_epoch\': 1716072393, \'localtime\': \'2024-05-18 17:46\'}, \'current\': {\'last_updated_epoch\': 1716072300, \'last_updated\': \'2024-05-18 17:45\', \'temp_c\': 29.0, \'temp_f\': 84.2, \'is_day\': 1, \'condition\': {\'text\': \'Partly cloudy\', \'icon\': \'//cdn.weatherapi.com/weather/64x64/day/116.png\', \'code\': 1003}, \'wind_mph\': 6.9, \'wind_kph\': 11.2, \'wind_degree\': 220, \'wind_dir\': \'SW\', \'pressure_mb\': 1011.0, \'pressure_in\': 29.86, \'precip_mm\': 0.0, \'precip_in\': 0.0, \'humidity\': 46, \'cloud\': 50, \'feelslike_c\': 31.4, \'feelslike_f\': 88.6, \'vis_km\': 16.0, \'vis_miles\': 9.0, \'uv\': 7.0, \'gust_mph\': 7.4, \'gust_kph\': 11.9}}"}]', 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'tool', 'name': 'tavily_search_results_json', 'id': '728f8ac9-729e-4bf7-b560-b332a73c8f47', 'tool_call_id': 'toolu_01NGhKmeciaT7TfhBSwUT3mi'}]}}



Receiving new event of type: data...
{'agent': {'messages': [{'content': [{'text': 'The search results seem to be for the weather in Louisiana, Missouri rather than Los Angeles, California. Let me try the search again:', 'type': 'text'}, {'id': 'toolu_019YAXWMK33tG9DaxMzrowc8', 'input': {'query': 'weather in los angeles california'}, 'name': 'tavily_search_results_json', 'type': 'tool_use'}], 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'ai', 'name': None, 'id': 'run-c42a3b14-2611-4a1d-8907-95dcdb18f07f', 'example': False, 'tool_calls': [{'name': 'tavily_search_results_json', 'args': {'query': 'weather in los angeles california'}, 'id': 'toolu_019YAXWMK33tG9DaxMzrowc8'}], 'invalid_tool_calls': []}]}}



Receiving new event of type: end...
None

Edit an old state

Let's now imagine we want to go back in time and edit the tool call after we had already made it. In order to do this, we can get first get the full history of the thread.

In [46]:

thread_history = await client.threads.get_history(thread['thread_id'], limit=100)

In [47]:

len(thread_history)

Out [47]:

11

After that, we can get the correct state we want to be in. The 0th index state is the most recent one, while the -1 index state is the first. In this case, we want to go to the state where the last message had the tool calls for weather in los angeles

In [48]:

rewind_state = thread_history[3]
rewind_state['values']['messages'][-1]['tool_calls']

Out [48]:

[{'name': 'tavily_search_results_json',
  'args': {'query': 'weather in los angeles'},
  'id': 'toolu_01FnuDKhUfagwoqhNfiTYTfS'}]

In [49]:

rewind_state['config']

Out [49]:

{'configurable': {'thread_id': 'df85453d-cb86-48c8-ae84-12081faa1bdf',
  'thread_ts': '1ef15582-3442-6db7-8006-9166bbb0e80f'}}

If we want to, we can now resume execution from that place in time

In [50]:

input = None
async for chunk in client.runs.stream(
    thread['thread_id'], 
    assistant['assistant_id'], 
    input=input, 
    stream_mode="updates", 
    interrupt_before=['action'],
    config=rewind_state['config']
):
    print(f"Receiving new event of type: {chunk.event}...")
    print(chunk.data)
    print("\n\n")

Receiving new event of type: metadata...
{'run_id': 'a1cc9263-ef0a-4c04-9194-6f01624d0ef0'}



Receiving new event of type: data...
{'action': {'messages': [{'content': '[{"url": "https://www.weatherapi.com/", "content": "{\'location\': {\'name\': \'Los Angeles\', \'region\': \'California\', \'country\': \'United States of America\', \'lat\': 34.05, \'lon\': -118.24, \'tz_id\': \'America/Los_Angeles\', \'localtime_epoch\': 1716071728, \'localtime\': \'2024-05-18 15:35\'}, \'current\': {\'last_updated_epoch\': 1716071400, \'last_updated\': \'2024-05-18 15:30\', \'temp_c\': 20.0, \'temp_f\': 68.0, \'is_day\': 1, \'condition\': {\'text\': \'Partly cloudy\', \'icon\': \'//cdn.weatherapi.com/weather/64x64/day/116.png\', \'code\': 1003}, \'wind_mph\': 2.2, \'wind_kph\': 3.6, \'wind_degree\': 226, \'wind_dir\': \'SW\', \'pressure_mb\': 1016.0, \'pressure_in\': 29.99, \'precip_mm\': 0.0, \'precip_in\': 0.0, \'humidity\': 61, \'cloud\': 50, \'feelslike_c\': 20.0, \'feelslike_f\': 68.0, \'vis_km\': 16.0, \'vis_miles\': 9.0, \'uv\': 6.0, \'gust_mph\': 12.6, \'gust_kph\': 20.3}}"}]', 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'tool', 'name': 'tavily_search_results_json', 'id': '7137b2e5-566b-418b-b642-b3c6b64c5224', 'tool_call_id': 'toolu_01FnuDKhUfagwoqhNfiTYTfS'}]}}



Receiving new event of type: data...
{'agent': {'messages': [{'content': 'The search results show the current weather conditions in Los Angeles. As of 3:30pm on May 18, 2024, the weather in Los Angeles is partly cloudy with a temperature around 68°F (20°C). Winds are light from the southwest around 2-3 mph. The humidity is 61% and visibility is good at 9 miles. Overall, it appears to be a nice spring day in LA with partly sunny skies and comfortable temperatures in the upper 60s.', 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'ai', 'name': None, 'id': 'run-3966b68a-c381-4933-a852-e6a4697c962c', 'example': False, 'tool_calls': [], 'invalid_tool_calls': []}]}}



Receiving new event of type: end...
None

In [ ]: