WEBDIENSTE/langgraph
1
0
Fork 0
mirror of https://github.com/langchain-ai/langgraph.git synced 2026-10-19 14:31:47 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
1ffa5bc7406cb80fffa7590438e8c55d44aa4b7d
langgraph/libs/sdk-py/tests/streaming/test_lifecycle_watcher.py
T
Elior Nataf LackritzandGitHub 9e6fc45c6f fix(sdk-py): surface thread stream errors and stop retrying a rejected stream (#9317) 
## Summary

Thread streams in `langgraph_sdk` now report why they ended. An `id:
null` error frame becomes the error `thread.output` raises, the sync
client no longer blocks forever when the server closes the stream, and a
stream the server rejects with a 4xx ends at once instead of being
retried.

## Problem

- A server sends `{"type": "error", "id": null, ...}` just before it
closes a thread stream whose read failed. The SDK dropped the frame, so
`thread.output` raised a generic "lifecycle stream ended before terminal
event" (async) or blocked forever (sync: the lifecycle watcher returned
without settling the run).
- A 403 or 404 on the stream request was retried like a network error,
then reported as `RuntimeError("Lifecycle transport failed: ...")` with
no status. A server that rejects a WebSocket after accepting it closes
with 4000 plus the HTTP status (4403, 4404, 4422), since a browser can't
read a refused upgrade's status. The shared stream counted each such
reconnect as a success and reconnected forever.

## Fix

- The lifecycle watcher fails the run with the frame's error on a clean
close (both clients). The shared stream logs the frame instead of
dropping it: projections still end when the stream closes, as before,
and `thread.output` carries the error.
- A 4xx other than 408 and 429, as a status or as a 4000+status close
code, ends the stream without retries. `thread.output` now raises that
`httpx.HTTPStatusError` or `websockets.exceptions.ConnectionClosedError`
instead of the generic `RuntimeError`.
- For a minted `thread_id`, the lifecycle watcher starts after
`run.start` creates the thread. Without this, the 4xx stop would turn a
stream opened under custom auth for a not-yet-created thread from
retried into failed. The sync client starts it under a lock, since
`run.start` can be called from several threads.
- Drops the first of two `SyncStreamController._reconnect_shared_stream`
definitions. It was shadowed by the second and never ran.

Not in this PR: over WebSocket the server closes with 1011 after an
error frame, which is still retried, while over SSE the close is clean
and fails the stream. Reconnecting after a clean close, as
`langchain-ai/langgraphjs#2812` does for the JS SDK, is a separate
change.

## Test plan

- [x] The 10 new behavior tests (async and sync: error frame, clean
close, 404, 4404 close, reconnect on 404, minted `thread_id`) fail on
`main` and pass here; a race test checks that two concurrent `run.start`
calls start one watcher.
- [x] `make test` in `libs/sdk-py`: 541 passed. `make lint` clean,
`check_sdk_methods.py` passes.
- [x] `RemoteGraph` v3 tests in `libs/langgraph`
(`test_remote_graph_v3.py`,
`test_stream_events_v3_kwarg_forwarding.py`): 94 passed against this
SDK.
- [x] Against a local server with custom auth: a thread the caller can't
read fails at once with 404 (SSE) or 4404 (WebSocket), and a new thread
works with a 1 s pause before `run.start`, which fails on `main`.
2026-10-16 15:29:34 -04:00

412 lines
16 KiB
Python
Raw Blame History

"""Tests for the lifecycle watcher: `interrupted` / `interrupts` state."""
from __future__ import annotations
import asyncio
import contextlib
from typing import Any
import httpx
import pytest
from langgraph_sdk._async.http import HttpClient
from langgraph_sdk._async.threads import ThreadsClient
from langgraph_sdk.stream.transport import EventStreamHandle, ProtocolSseTransport
from streaming._events import (
input_requested_event,
lifecycle_completed_event,
lifecycle_errored_event,
lifecycle_event,
thread_stream_failed_frame,
)
from streaming._fake_server import FakeServer, _StreamScript
async def test_interrupted_starts_false():
async with httpx.AsyncClient(base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
assert thread.interrupted is False
assert thread.interrupts == []
async def test_interrupts_populated_from_input_requested_event():
fake = FakeServer()
fake.script([input_requested_event(seq=0)])
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
await thread.run.start(input={})
# Lifecycle watcher consumes asynchronously — poll briefly.
for _ in range(20):
if thread.interrupted:
break
await asyncio.sleep(0.05)
assert thread.interrupted is True
assert len(thread.interrupts) == 1
assert thread.interrupts[0]["interrupt_id"] == "i-1"
async def test_aenter_starts_lifecycle_watcher():
"""Entering with an explicit thread_id opens the lifecycle/input SSE before run.start."""
fake = FakeServer()
fake.script([lifecycle_event(seq=0, phase="started")])
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
# The lifecycle watcher task must be created on __aenter__, no run.start needed.
assert thread._lifecycle_watcher_task is not None
# Poll until the watcher has consumed the started event.
for _ in range(20):
if thread._run_seen:
break
await asyncio.sleep(0.05)
assert thread._run_seen is True
# No run.start was ever called — but the server still received a stream request.
assert len(fake.stream_request_bodies) >= 1
async def test_reattach_observes_terminal_state():
"""Reattach (no run.start) consumes lifecycle replay and observes terminal state."""
fake = FakeServer()
fake.script(
[
lifecycle_event(seq=0, phase="running"),
lifecycle_event(seq=1, phase="completed"),
]
)
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="existing", assistant_id="agent") as thread:
# Never call run.start — this is a reattach scenario.
# Poll until _run_done is resolved.
for _ in range(20):
run_done = thread._run_done
if run_done is not None and run_done.done():
break
await asyncio.sleep(0.05)
assert thread._run_done is not None
assert thread._run_done.done()
terminal = thread._run_done.result()
assert terminal.status == "completed"
assert terminal.error is None
async def test_terminal_lifecycle_clears_interrupts():
"""Terminal lifecycle event clears interrupted/interrupts."""
fake = FakeServer()
fake.script([lifecycle_event(seq=0, phase="completed")])
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
# Manually set interrupted state to simulate a prior interrupt.
thread.interrupted = True
thread.interrupts = [
{"interrupt_id": "i-1", "value": None, "namespace": []}
]
# Poll until the lifecycle watcher processes the completed event.
for _ in range(20):
if not thread.interrupted:
break
await asyncio.sleep(0.05)
assert thread.interrupted is False
assert thread.interrupts == []
async def test_subgraph_completed_event_does_not_end_run():
fake = FakeServer()
fake.script(
[
lifecycle_completed_event(seq=0, namespace=["child:1"]),
lifecycle_errored_event(seq=1, error="root failed"),
]
)
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
run_done = thread._run_done
assert run_done is not None
terminal = await asyncio.wait_for(run_done, timeout=2.0)
assert terminal.status == "errored", "a subgraph's completed event ended the run"
assert "root failed" in str(terminal.error)
async def test_lifecycle_error_captured_for_output():
"""Lifecycle error terminal state is captured in _run_done with error set."""
fake = FakeServer()
fake.script([lifecycle_event(seq=0, phase="errored", error="something went wrong")])
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
# Poll until _run_done is resolved.
for _ in range(20):
run_done = thread._run_done
if run_done is not None and run_done.done():
break
await asyncio.sleep(0.05)
assert thread._run_done is not None
assert thread._run_done.done()
terminal = thread._run_done.result()
assert terminal.status == "errored"
assert terminal.error is not None
assert "something went wrong" in str(terminal.error)
async def test_run_start_sets_run_seen():
"""run.start() sets _run_seen to True (even without lifecycle event)."""
fake = FakeServer()
fake.script([]) # No events; the command response is sufficient.
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
assert thread._run_seen is False
await thread.run.start(input={})
# _run_seen is set synchronously in run.start, before awaiting the result.
assert thread._run_seen is True
async def test_lifecycle_clean_eof_resolves_run_done_with_errored():
"""If the lifecycle SSE stream ends cleanly (server closes without a
terminal `completed` or `errored` event), `_run_done` must resolve with
an errored terminal so awaiters don't hang."""
fake = FakeServer()
# Emit a non-terminal lifecycle event, then close cleanly without
# `completed` or `errored`.
fake.script([lifecycle_event(seq=0, phase="started")])
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
run_done = thread._run_done
assert run_done is not None
terminal = await asyncio.wait_for(run_done, timeout=2.0)
assert terminal.status == "errored"
assert terminal.error is not None
assert "ended before terminal" in str(terminal.error)
# Quiet unused-import warning under strict configs.
_ = pytest
async def test_lifecycle_mid_iteration_error_resolves_run_done_with_error(
monkeypatch: Any,
) -> None:
"""If the transport reports an error via `handle.done` after iteration
exits without a terminal lifecycle event, `_run_done` propagates the
transport error rather than the generic clean-EOF message."""
def synthetic_handle() -> EventStreamHandle:
loop = asyncio.get_running_loop()
ready: asyncio.Future[None] = loop.create_future()
ready.set_result(None)
done: asyncio.Future[BaseException | None] = loop.create_future()
done.set_result(RuntimeError("simulated transport error"))
async def empty_events() -> Any:
if False:
yield # pragma: no cover # make this an async generator
return
async def noop_close() -> None:
return
return EventStreamHandle(
events=empty_events(),
ready=ready,
done=done,
close=noop_close,
)
def patched_open(_self: ProtocolSseTransport, _params: Any) -> EventStreamHandle:
return synthetic_handle()
monkeypatch.setattr(ProtocolSseTransport, "open_event_stream", patched_open)
fake = FakeServer()
fake.script([])
asgi = httpx.ASGITransport(app=fake.app)
async with httpx.AsyncClient(transport=asgi, base_url="http://test") as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
run_done = thread._run_done
assert run_done is not None
terminal = await asyncio.wait_for(run_done, timeout=2.0)
assert terminal.status == "errored"
assert terminal.error is not None
assert "simulated transport error" in str(terminal.error)
# Quiet unused-import warnings under strict configs.
_ = contextlib
async def test_lifecycle_watcher_reconnects_with_since_after_transport_drop():
fake = FakeServer()
fake.set_state({"ok": True})
fake.script_sequence(
[
_StreamScript(
events=[lifecycle_event(seq=1, phase="running")],
fail_after=1,
),
_StreamScript(events=[lifecycle_completed_event(seq=2)]),
]
)
async with httpx.AsyncClient(
transport=fake.transport, base_url="http://test"
) as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="existing", assistant_id="agent") as thread:
for _ in range(20):
run_done = thread._run_done
if run_done is not None and run_done.done():
break
await asyncio.sleep(0.05)
assert thread._run_done is not None
terminal = thread._run_done.result()
assert terminal.status == "completed"
assert terminal.error is None
assert fake.stream_request_bodies[0]["channels"] == ["lifecycle", "input"]
assert fake.stream_request_bodies[1]["channels"] == ["lifecycle", "input"]
assert fake.stream_request_bodies[1]["since"] == 1
async def test_lifecycle_error_frame_becomes_the_run_error():
fake = FakeServer()
fake.script([lifecycle_event(seq=0, phase="running"), thread_stream_failed_frame()])
async with httpx.AsyncClient(
transport=fake.transport, base_url="http://test"
) as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
run_done = thread._run_done
assert run_done is not None
terminal = await asyncio.wait_for(run_done, timeout=2.0)
assert terminal.status == "errored"
assert (
str(terminal.error)
== "Protocol error [unknown_error]: Thread event stream failed."
)
async def test_lifecycle_watcher_stops_on_a_404_without_retrying():
fake = FakeServer()
fake.script_sequence([_StreamScript(events=[], status=404)])
async with httpx.AsyncClient(
transport=fake.transport, base_url="http://test"
) as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-1", assistant_id="agent") as thread:
run_done = thread._run_done
assert run_done is not None
terminal = await asyncio.wait_for(run_done, timeout=2.0)
assert isinstance(terminal.error, httpx.HTTPStatusError)
assert terminal.error.response.status_code == 404
assert len(fake.stream_request_bodies) == 1
async def test_minted_thread_opens_the_lifecycle_stream_after_run_start():
fake = FakeServer()
fake.script([lifecycle_completed_event(seq=1)])
async with httpx.AsyncClient(
transport=fake.transport, base_url="http://test"
) as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(assistant_id="agent") as thread:
assert thread._lifecycle_watcher_task is None
await thread.run.start(input={})
run_done = thread._run_done
assert run_done is not None
terminal = await asyncio.wait_for(run_done, timeout=2.0)
assert terminal.status == "completed"
def _new_thread_scripts() -> list[_StreamScript]:
return [
_StreamScript(events=[], status=404),
_StreamScript(events=[lifecycle_completed_event(seq=1)]),
]
async def test_run_start_restarts_a_watcher_stopped_by_a_404_for_a_new_thread():
fake = FakeServer()
fake.script_sequence(_new_thread_scripts())
fake.set_state({"ok": True})
async with httpx.AsyncClient(
transport=fake.transport, base_url="http://test"
) as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-new", assistant_id="agent") as thread:
rejected = thread._run_done
assert rejected is not None
await asyncio.wait_for(rejected, timeout=2.0)
await thread.run.start(input={})
output = await thread.output.with_timeout(2.0)
assert output == {"ok": True}
assert len(fake.stream_request_bodies) == 2
async def test_lifecycle_watcher_retries_a_404_that_arrives_after_run_start():
fake = FakeServer()
fake.script_sequence(_new_thread_scripts())
fake.set_state({"ok": True})
stream_opened = asyncio.Event()
run_started = asyncio.Event()
async def handle(request: httpx.Request) -> httpx.Response:
if (
request.url.path.endswith("/stream/events")
and not fake.stream_request_bodies
):
stream_opened.set()
await run_started.wait()
return await fake._handle_request(request)
async with httpx.AsyncClient(
transport=httpx.MockTransport(handle), base_url="http://test"
) as raw:
threads = ThreadsClient(HttpClient(raw))
async with threads.stream(thread_id="t-new", assistant_id="agent") as thread:
await asyncio.wait_for(stream_opened.wait(), timeout=2.0)
await thread.run.start(input={})
run_started.set()
output = await thread.output.with_timeout(2.0)
assert output == {"ok": True}
assert len(fake.stream_request_bodies) == 2
async def test_lifecycle_watcher_reports_the_error_frame_when_reconnects_run_out():
fake = FakeServer()
async with httpx.AsyncClient(
transport=fake.transport, base_url="http://test"
) as raw:
threads = ThreadsClient(HttpClient(raw))
stream = threads.stream(thread_id="t-1", assistant_id="agent")
stream._lifecycle_max_reconnect_attempts = 1
fake.script_sequence(
[
_StreamScript(events=[thread_stream_failed_frame()], fail_after=1)
for _ in range(stream._lifecycle_max_reconnect_attempts + 1)
]
)
async with stream as thread:
run_done = thread._run_done
assert run_done is not None
terminal = await asyncio.wait_for(run_done, timeout=2.0)
assert (
str(terminal.error)
== "Protocol error [unknown_error]: Thread event stream failed."
)
Reference in New Issue View Git Blame Copy Permalink