Fixes langchain-ai/deepagents#3774
Reworks the fresh-thread `update_state` fix for `DeltaChannel`: instead
of creating stub checkpoint (#8011), force a new Snapshot into the first
checkpoint so the value is stored inline and needs no ancestor replay.
## Background
`update_state` / `bulk_update_state` on a *fresh* thread silently
dropped the first write to a `DeltaChannel`.
By design, a `DeltaChannel` reconstructs its value by walking ancestor
checkpoints and replaying the writes attached to them.
Checkpoint writes need a parent to persist. But on a fresh thread there
is no ancestor, there is no parent to use.
#8011 fixed this by lazily persisting an empty stub checkpoint (step
`-1`) to give the first write a parent to use. This PR reverts that and
takes a simpler route.
## A better fix
On a fresh thread (`saved is None`), force a snapshot of every available
`DeltaChannel` into the first checkpoint via `create_checkpoint(...,
channels_to_snapshot=...)`.
This way, the read/replay path is untouched and no stub is needed.
## Behavior change
A fresh-thread `update_state` now produces a **single** self-contained
checkpoint (step `0`, no parent, snapshot inline) instead of two (stub
step `-1` + update step `0`). This is visible via `get_state_history`.
## Verify
`make format`, `make lint`, `make test` in `libs/langgraph`.
`tests/test_delta_channel_update_state.py` is updated to assert the
single-checkpoint shape on a fresh thread and pins the non-fresh paths
(`update_state` after `invoke`, consecutive `update_state`,
`bulk_update_state`) against regression.
Fix exit-mode DeltaChannel `PutWrites` task IDs so they remain valid RFC
UUIDs while preserving superstep ordering for `ORDER BY task_id, idx`.
Follow-up to #7730: `_put_exit_delta_writes` used `f"{step:08d}-{tid}"`,
which produces a 6-segment string Postgres rejects for
`checkpoint_writes.task_id uuid` (e.g. `invalid input syntax for type
uuid`) in LangGraph-API.
This new `exit_delta_task_id()` helper embeds the superstep in the first
UUID group and keeps the original task UUID in the remaining segments
(e.g. `00000001-0270-bf16-1ef8-fb321bef9f3d`).
**Breaking changes:** None.
**Verification:**
- `cd libs/langgraph && make format && make lint && make test
TEST="tests/test_delta_channel_exit_mode.py"` (12 passed)
- New unit test asserts synthetic IDs parse as UUID, sort by superstep,
and that the old format is invalid
---------
Signed-off-by: Quanzheng Long <prclqz@gmail.com>
Co-authored-by: open-swe[bot] <215916821+open-swe[bot]@users.noreply.github.com>
Add a standalone recovery tool at `examples/delta-channel-dump/` for
operators rolling back from langgraph >= 1.2 / deepagents 0.6.x to an
older runtime that does not understand `EXT_DELTA_SNAPSHOT` msgpack
blobs. Without this, channels like `messages` can appear empty after
rollback because the old reducer does not decode delta snapshots.
`dump.py` connects directly to Postgres, walks the checkpoint parent
chain (mirroring `aget_delta_channel_history`), decodes msgpack blobs
via `ormsgpack` with a hand-rolled EXT 0–7 hook (no langgraph imports),
and emits JSON with per-channel `seed` + oldest-to-newest `writes`. The
operator reduces and re-applies via `update_state` manually.
## Scope
- Postgres checkpointer only
- Deps: `psycopg[binary]`, `ormsgpack` (not added to repo
`pyproject.toml` — operator installs at runtime)
- No reducer application, no DB writes, no AES/custom encryption support
(fails loudly on encrypted blob types)
## How verified
- E2E against local Postgres: 15 runs of `delta_channel_messages_freq`
(`snapshot_frequency=10`)
- Asserted `delta_kind == "snapshot"`, `len(seed) == 10`, `len(writes)
== 5`, message ids `ai-0`..`ai-14`
- Confirmed portability: fresh venv with only `pip install
"psycopg[binary]" ormsgpack` produced identical output
Fixeslangchain-ai/deepagents#3774
## Summary
`Pregel.update_state` / `aupdate_state` on a fresh thread silently
dropped the first write to a `DeltaChannel`-backed channel (e.g.
`DeepAgentState.messages`). This PR persists the first write under a
lazily-created stub checkpoint so the read-path ancestor walk can replay
it.
## Root cause
`DeltaChannel` reads its value back by walking ancestor checkpoints and
replaying writes attached to them — non-snapshot steps don't store the
value in `channel_values`. In `bulk_update_state` the channel writes
were only persisted via `checkpointer.put_writes(...)` when a previous
checkpoint existed:
```python
channel_writes = [w for w in task.writes if w[0] != PUSH]
if saved and channel_writes:
checkpointer.put_writes(checkpoint_config, channel_writes, task_id)
```
On a fresh thread `saved is None`, so the `if saved` guard skipped
persistence entirely. `create_checkpoint` then bumped the channel
version but stored neither a value nor replayable writes, so reads
returned `[]`.
## Fix
In both `bulk_update_state` (sync) and `abulk_update_state` (async),
when the thread has no persisted parent **and** at least one write
targets a `DeltaChannel`, lazily persist an empty stub checkpoint and
use it as the parent for both the channel writes and the new update
checkpoint. Mirrors the existing exit-mode pattern in
`_loop._put_exit_delta_writes`.
The behavior for non-delta writes on a fresh thread is preserved (skip
`put_writes` — values are stored directly in the new checkpoint's
`channel_values`), so non-delta `update_state` paths add no extra
checkpoint rows.
## Test coverage
New tests in `libs/langgraph/tests/test_delta_channel_update_state.py`
(9 tests, sync + async):
- **Fresh-thread regression** (the bug): single `update_state` writes a
message and reads back via `get_state`. Without the fix, both sync and
async fail with `assert [] == ['hello']`.
- **`update_state` after `invoke`**: pins down the previously-working
non-fresh-thread path so the lazy-stub change doesn't regress it.
- **Consecutive `update_state`s**: second call sees a real parent
(`saved is not None`) and takes the original write path; both messages
round-trip in chronological order.
- **Update-by-id end-to-end via `update_state`**:
`_messages_delta_reducer`'s dedup-by-id semantics work through the
`update_state` path, not just `invoke`.
- **`bulk_update_state` with multiple per-superstep updates**: locks in
the per-task `put_writes` loop so all task writes persist (not just the
last task's).
- **State-history chain shape**: validates the lazy stub via the public
API — `get_state_history` returns `[update_checkpoint, stub]` where the
stub has `source='update'`, `step=-1`, no parent, and the update
checkpoint's `parent_config` points at the stub.
## Verification
- All 9 tests in `tests/test_delta_channel_update_state.py` pass.
- All 4 existing delta-channel suites pass
(`test_delta_channel_exit_mode.py`, `test_delta_channel_migration.py`,
`test_delta_channel_id_stability.py`,
`test_delta_channel_supersteps_bound.py` — 30 tests, 39 total with the
new file).
- All `update_state`-related tests across `test_pregel`,
`test_pregel_async`, `test_time_travel`, `test_time_travel_async` pass
(10 tests).
- `make format`, `make lint`, full `make test` pass locally in
`libs/langgraph`.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
- **Consolidate error writes:** When a node fails and has an error
handler, `commit()` now appends both `ERROR` and `ERROR_SOURCE_NODE` in
a single `put_writes` call, eliminating the redundant overwrite that
`schedule_error_handler` used to do.
- **Ensure durability before handler execution:** Reuses the
`_delta_write_futs` pattern — a new `_error_handler_write_futs` list
collects the persistence future from `put_writes` when
`ERROR_SOURCE_NODE` is written, and `schedule_error_handler` /
`aschedule_error_handler` drain it (sync: `concurrent.futures.wait`,
async: `asyncio.gather`) before preparing the handler task.
- **Resume directly to error handler:** Adds
`_resume_error_handlers_if_applicable()` to `PregelLoop`, called from
`tick()` after `_reapply_writes_to_succeeded_nodes()`. On resume, it
detects `ERROR_SOURCE_NODE` markers in `checkpoint_pending_writes`,
marks the original task as done (so the runner skips it), and schedules
a fresh handler task.
- **Rename internal methods for clarity:** `_match_writes` →
`_reapply_writes_to_succeeded_nodes` (makes it clear that
failed/interrupted tasks are skipped); `_resume_error_handlers` →
`_resume_error_handlers_if_applicable`.
## Test plan
- [x] `test_error_handler_resumes_after_crash`: single node fails,
handler crashes, resume re-runs the handler (not the original node).
Verifies `NodeError.node` and error content survive checkpoint
round-trip.
- [x] `test_error_handler_resumes_after_crash_multiple_nodes`: two nodes
fail concurrently in the same superstep, each with its own handler.
Verifies error handler starts while other nodes are still in-flight (via
`threading.Event`), and on resume both handlers re-run with correct
`NodeError.node` and error content.
- [x] All 101 tests in `test_retry.py` pass (including 19 error-handler
tests).
- [x] `make format` + `make lint` clean.
## Summary
Add a system-wide upper bound on supersteps-since-last-snapshot for
`DeltaChannel`, preventing unbounded ancestor walks on long-lived
threads where a delta channel stops receiving writes.
**Problem:** If a delta channel is written a few times (below
`snapshot_frequency`) and then never written again, it is never
snapshotted. Every subsequent run triggers an ancestor walk that grows
linearly with thread length — on long threads this becomes catastrophic.
**Solution:** Track a second counter (total supersteps) per delta
channel alongside the existing update count. Force a snapshot when
EITHER `updates >= snapshot_frequency` OR `supersteps >=
DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT` (default 5000, overridable via env
`LANGGRAPH_DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT`).
### Changes
- **`checkpoint` lib**: Rename metadata field
`delta_updates_since_snapshot: dict[str, int]` ->
`counters_since_last_snapshot: dict[str, tuple[int, int]]` where index 0
= updates, index 1 = supersteps.
- **`langgraph/_internal/_config.py`**: Add
`DELTA_MAX_SUPERSTEPS_SINCE_SNAPSHOT` constant with env override.
- **`langgraph/pregel/_checkpoint.py`**: Update
`delta_channels_to_snapshot()` predicate to fire on either threshold.
Rename reader helper to `read_counters_since_last_snapshot()`.
- **`langgraph/pregel/_loop.py`**: Iterate all delta channels each
superstep (not just updated ones) to bump the supersteps counter. Reset
both counters to `(0, 0)` on snapshot.
- **Tests**: Updated existing exit-mode tests for new field shape. Added
4 new tests covering forced snapshot (single run + multi-run
accumulation), predicate unit test, and counter reset.
## Test plan
- [x] `test_delta_channel_supersteps_bound.py` — 4 new tests all pass
- [x] `test_delta_channel_exit_mode.py` — 11 existing tests updated and
pass
- [x] `test_delta_channel_migration.py` — 11 tests pass
- [x] `test_channels.py` — 29 tests pass
- [x] `test_pregel.py` — 457 tests pass
- [x] `libs/checkpoint` test suite — 151 pass, 16 skipped
- [x] `make lint` clean (langgraph + checkpoint)
## Summary
Add a user-facing design doc and `get_delta_channel_keepset` helper for
third-party `BaseCheckpointSaver` authors who need to support graphs
using `DeltaChannel`.
**Deliverables:**
1. ~~**`docs/delta-channel-checkpointer-guide.md`** — comprehensive
guide covering~~:
moved to docs repo
2. **`BaseCheckpointSaver.get_delta_channel_keepset` /
`aget_delta_channel_keepset`** — returns the minimum set of ancestor
`checkpoint_id`s that must survive deletion for a given head's
`DeltaChannel` reconstruction to remain intact. Enables safe `prune`
implementations without silently corrupting delta history.
3. **Docstring warnings** on `prune`, `aprune`, `delete_for_runs`,
`adelete_for_runs`, `copy_thread`, `acopy_thread` explaining the
DeltaChannel pitfall (silent data loss if ancestor writes/snapshots are
deleted).
4. **Three new conformance capabilities** in
`libs/checkpoint-conformance`:
- `delta_channel_history` — validates the `aget_delta_channel_history`
walk contract
- `delta_channel_keepset` — validates the keep-set contract
- `delta_channel_reconstruction` — end-to-end round-trip (aput +
aput_writes + history + reconstruct)
## Test plan
- [x] `make format lint` passes in `libs/checkpoint`,
`libs/checkpoint-conformance`
- [x] All three new conformance capabilities pass against
`InMemorySaver`
- [x] Run conformance against SQLite saver
- [x] Run conformance against Postgres saver
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
De-flake
`test_arun_with_retry_timeout_observer_emits_progress_on_heartbeat` —
the test was hitting a CI-runner-load-sensitive race where the
idle-timeout watchdog could fire before the task body's first await ran.
## Root cause
`_TimedAttemptScope.__init__` sets `_last_progress = time.monotonic()`
immediately, but the watchdog itself doesn't start polling until *after*
`wrap_config` and task scheduling. Under heavy CI load that gap can grow
large enough that:
```
T₀ scope.__init__() → _last_progress = T₀
… some scheduling slack …
Tₙ watchdog runs, computes remaining = T₀ + 0.2 − Tₙ ≤ 0 → TimeoutError fires
```
The error reports `elapsed: 0.000s` because `elapsed` is measured from
the post-scheduling `start` (≈Tₙ), not from `_last_progress` (T₀). The
previous test set `idle_timeout=0.2s`, which left almost no headroom for
that scheduling slack.
## Fix (test-side only — no production change)
- **Heartbeat at task-body entry**: `runtime.heartbeat()` is now called
before the first `await asyncio.sleep(...)`, which resets
`_last_progress` to "now" the moment the task body actually starts
running. This eliminates the scope-init-to-first-await gap as a flake
source.
- **Idle timeout 0.2s → 1.0s**: gives ~5× headroom over the ~400ms task
duration, so scheduling pressure stays comfortably within budget.
## Why test-side instead of fixing the production race
The proper production fix would be to set `_last_progress` at
watchdog-entry time rather than at scope-init time. That's a behaviour
change in the retry/timeout machinery and out of scope for a flaky-test
fix. The two test-side defenses make this particular test stable without
touching production semantics; the underlying race in
`_TimedAttemptScope` is worth a separate follow-up.
## Test plan
- 10/10 repeated local runs pass:
```
uv run pytest
tests/test_retry.py::test_arun_with_retry_timeout_observer_emits_progress_on_heartbeat
--count=10
```
- All assertions still meaningful: still verifies start/finish events,
at least one progress event, rate-limited progress count (≤ total
events), and per-event metadata (task_name, attempt, idle_timeout_secs,
progress_at).
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Replaces `durability="exit"`'s blanket force-snapshot of every
`DeltaChannel` with proper write persistence that honors per-channel
`snapshot_frequency`, plus closes two latent bugs the force-snapshot was
masking.
Before: every exit-mode run wrote a full `_DeltaSnapshot` blob for every
delta channel, even when the channel had zero updates this run and was
nowhere near its `snapshot_frequency`. After: the same count-based
decision used by `durability="sync"`/`"async"` applies — channels at or
above `snapshot_frequency` snapshot; channels below it persist their
accumulated writes via a lazy "stub" anchor; untouched channels write
nothing.
## What changed
**Core redesign** (`pregel/_loop.py`, `pregel/_checkpoint.py`)
- Drop `force_delta_snapshot` from `create_checkpoint` and
`_should_snapshot_delta`.
- Add `decide_delta_snapshots(channels, counts)` pure helper used by
both `create_checkpoint` and the new exit-mode peek-ahead path.
- Add `_exit_delta_writes` accumulator: every delta-channel write
produced during a `durability="exit"` run (input writes from `_first` +
per-superstep writes captured before `pending_writes.clear()` in
`after_tick`) is collected into this list.
- Add `_put_exit_delta_writes` (sync + async): runs from
`_suppress_interrupt` BEFORE `_put_checkpoint(exiting=True)`. Filters
out channels that will snapshot, then persists remaining writes to
`checkpoint_writes` under an anchor parent. The anchor is the existing
saved parent on resumed runs, or a lazily-created empty stub on first
runs.
- Visibility ordering: stub put goes onto `_put_checkpoint_fut` (becomes
the next put's `prev`); exit-write futures go onto `_delta_write_futs`.
The existing `_checkpointer_put_after_previous` already drains both
before calling `saver.put`, so `final_checkpoint` is structurally
guaranteed to land last — readers never see a partial view.
**Latent bugs fixed (previously masked by force-snapshot)**
- **Sync drain race**: `SyncPregelLoop` now initializes
`_delta_write_futs = []` in `__enter__` and drains it in sync
`_checkpointer_put_after_previous` before `put`, mirroring the async
version. Without this, a multi-worker `BackgroundExecutor` could publish
a checkpoint before the writes that produced it.
- **Count double-bump in exit mode**: in `_put_checkpoint`,
`delta_updates_since_snapshot` was being incremented twice for the last
superstep — once by the intermediate `after_tick` call, once by
`_suppress_interrupt`. Force-snapshot used to reset all counts to 0 so
this never persisted; without it, snapshots would fire one superstep
early after every exit-mode run. Fixed by gating the count-bump behind
`not exiting`.
**Pre-existing input-durability gap**
- In the plain (non-Command) input path of `_first`, delta-channel input
writes are now persisted via `put_writes` (mirroring the Command path),
so sub-frequency inputs survive a `get_state` on resumed runs in
`sync`/`async` durability. Note: first-run `sync`/`async` still has the
same gap (writes orphan on the synthetic-empty parent id). That's
flagged as a follow-up — out of scope for this PR.
## Test plan
- Existing `tests/test_pregel.py` and `tests/test_pregel_async.py` pass
unchanged.
- Existing `tests/test_channels.py` (29 tests) and
`tests/test_delta_channel_migration.py` pass unchanged.
- New `tests/test_exit_delta_persistence.py` (11 tests) covers:
- **Write-path**: zero-write exit (no stub), all-snapshot first run (no
stub), sub-freq first run (single shared stub), sub-freq resumed run
(anchor on saved parent), sync-vs-exit count parity, mixed
snapshot/non-snapshot channels, snapshot fires at frequency.
- **Read-path**: K-run replay chain reads correctly across
stub→saved-parent transition; metadata `delta_updates_since_snapshot`
round-trips correctly; mixed sync/exit durability alternation produces
correct final state; snapshot+tail-deltas combination reads correctly.
- `make format && make lint && make test` in `libs/langgraph/`.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Sydney Runkle <sydneymarierunkle@gmail.com>
## Summary
Replaces the single-roundtrip `UNION ALL` DeltaChannel read with a
two-stage query that avoids fetching unused snapshot blobs, then removes
the old combined path entirely.
### Problem
`_get_channel_writes_history` used a single `UNION ALL` query that
fetched **all** checkpoint metadata, writes, and blobs for a
`(thread_id, channel)` in one shot. With `snapshot_frequency=N`, this
pulled back O(N/freq) full-size snapshot blobs even though only the
nearest one is needed to seed reconstruction. At 500 turns with
`snapshot_frequency=10`, this meant fetching ~100 complete
message-history snapshots per read.
### Solution
Two-stage read:
- **Stage 1** — lightweight scan of `checkpoints` only (no blob bytes):
walks the parent chain from the target checkpoint and stops at the first
ancestor with a snapshot, returning `chain_cids` and `seed_version`
- **Stage 2** — targeted fetch: only the writes for `chain_cids` and the
single seed blob at `seed_version`
The two-stage path is now unconditional — the old combined query and
`LG_DELTA_TWO_STAGE_QUERY` env-var gate have been removed.
### Sentinel cleanup
`DELTA_SENTINEL` is now a pure in-memory signal and is never written to
storage:
- Postgres `put()` already stripped it from `channel_values` before
writing blobs
- Memory saver `put()` now stores `"empty"` instead of serializing the
sentinel
- `EXT_DELTA_SENTINEL` (msgpack ext code 8) removed from
`JsonPlusSerializer`
- `DELTA_SENTINEL` is kept as an in-memory marker:
`DeltaChannel.checkpoint()` returns it so savers know to skip it, and
`_ChannelWritesHistory.seed` uses it to mean "no snapshot found, start
from empty"
## Performance
Benchmarked at `snapshot_frequency=10` on Postgres (`~100 tok/msg`):
| turns | old combined query | two-stage |
|------:|-------------------:|----------:|
| 50 | 6.0ms | 2.8ms (2.1x faster) |
| 100 | 10.1ms | 5.6ms (1.8x faster) |
| 500 | **216.1ms** | 15.3ms (**14x faster**) |
The old query's read time grew super-linearly with turn count because
each read fetched O(N/freq) full snapshot blobs. Two-stage keeps read
depth bounded by `snapshot_frequency` regardless of thread length.
## Test plan
- `make test` in `libs/checkpoint`, `libs/checkpoint-postgres`,
`libs/langgraph`
- Removed `test_delta_sentinel_serde_round_trip` (sentinel no longer
serializable)
- Updated `test_memory.py` — delta channel blobs stored as `"empty"`,
not serialized sentinel
- Updated `test_channels.py` — `channel_values` no longer contains
sentinel key for DeltaChannels
- Deleted `test_delta_channel_two_stage_benchmark.py` (one-stage vs
two-stage comparison; path no longer exists)
---------
Co-authored-by: Sydney Runkle <54324534+sydney-runkle@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## Summary
`NodeTimeoutError` previously inherited from `TimeoutError`, which is a
subclass of `OSError`. Since `OSError` is in the default `RetryPolicy`
blocklist, timeout errors from `TimeoutPolicy` were silently **not
retried** unless the user explicitly set `retry_on=NodeTimeoutError`.
This PR changes `NodeTimeoutError` to inherit from `Exception` directly,
so that the default `RetryPolicy` treats it as retryable — matching user
expectations when both `RetryPolicy` and `TimeoutPolicy` are configured
together.
- Change `NodeTimeoutError(TimeoutError)` →
`NodeTimeoutError(Exception)`
- Add test asserting `NodeTimeoutError` is retryable with the default
policy
- Add observer-ordering tests pinning down `finish=error` emission
timing relative to retry backoff, error handler start, and retry
exhaustion
## Breaking change
Code that catches `NodeTimeoutError` via `except TimeoutError` or
`except OSError` will no longer match. Use `except NodeTimeoutError`
instead.
## Test plan
- [x] `test_should_retry_default_retry_on` — asserts `NodeTimeoutError`
is retryable with default `RetryPolicy()`
- [x] Existing timeout+retry tests continue to pass (`test_retry.py`)
- [x] **Add tests and docs**:NA
- [x] **Lint and test**:
Additional guidelines:
- Make sure optional dependencies are imported within a function.
- Please do not add dependencies to `pyproject.toml` files (even
optional ones) unless they are **required** for unit tests.
- Most PRs should not touch more than one package.
- Changes should be backwards compatible.
- [x] **Add tests and docs**: If you're adding a new integration, you
must include:
1. A test for the integration, preferably unit tests that do not rely on
network access,
2. An example notebook showing its use. It lives in
`docs/docs/integrations` directory.
- [x] **Lint and test**: Run `make format`, `make lint` and `make test`
from the root of the package(s) you've modified. We will not consider a
PR unless these three are passing in CI. See [contribution
guidelines](https://docs.langchain.com/oss/python/contributing/overview)
for more.
Additional guidelines:
- Make sure optional dependencies are imported within a function.
- Please do not add dependencies to `pyproject.toml` files (even
optional ones) unless they are **required** for unit tests.
- Most PRs should not touch more than one package.
- Changes should be backwards compatible.
