## Summary
The sqlite delta history silently drops a parent checkpoint whose id sorts above its child's,
losing that parent's stored value and its pending writes. The channel hydrates short with no error.
Fixes#8550
## Problem
Stage 1 walked ancestors with:
```sql
WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id <= ?
ORDER BY checkpoint_id DESC
```
Ancestry is defined by the `parent_checkpoint_id` column. These two predicates add a second
requirement: that every child's id sorts above its parent's. The contract promises monotonic ids,
but that only holds within one process, so ids from processes with different clocks can break it.
When the requirement is violated the parent is excluded from the stream and its seed and writes go
with it. Dropping the range filter alone does not fix it: in `checkpoint_id DESC` order that parent
arrives *before* the target, so the walk streams past it before it has started.
## Fix
A recursive CTE anchored at the target, following `parent_checkpoint_id`:
```sql
WITH RECURSIVE ancestors(checkpoint_id, parent_checkpoint_id, type, checkpoint) AS (
SELECT ... FROM checkpoints
WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id = ?
UNION ALL
SELECT c.... FROM ancestors a CROSS JOIN checkpoints c
ON c.checkpoint_id = a.parent_checkpoint_id
WHERE c.thread_id = ? AND c.checkpoint_ns = ?
)
SELECT checkpoint_id, type, checkpoint FROM ancestors
```
Rows now arrive in walk order (target, parent, grandparent, ...), so `step_walk_with_row` no longer
needs its off-path skip or its `parent_cid` tracking; both are removed. The query reads only true
ancestors, where the old one read every row at or below the target including sibling branches.
`CROSS JOIN` pins the join order. The saver never runs `ANALYZE`, and with a plain `JOIN` sqlite put
`checkpoints` as the outer loop, scanning the whole thread on every recursion step. With `ancestors`
outside, each step is one primary key lookup. Through `get_delta_channel_history`:
| chain length | plain `JOIN` | `CROSS JOIN` |
| -- | -- | -- |
| 1000 | 0.032s | 0.001s |
| 2000 | 0.124s | 0.003s |
| 4000 | 0.475s | 0.006s |
## Cycle guard
Following pointers can loop where a bounded id scan could not, and a loop is reachable through
`put` alone: `put` writes with `INSERT OR REPLACE`, so re-putting an existing checkpoint id under a
descendant's config repoints that checkpoint at its own descendant. The walk stops on a repeated
`checkpoint_id` (one set insert per row, no depth ceiling that could truncate a long migrated
thread). sqlite yields recursive rows lazily, so abandoning the cursor ends the recursion.
`test_walk_terminates_when_put_makes_the_parent_chain_cycle` fails by hanging, not by asserting, if
the guard regresses (confirmed by deleting the guard locally). The package has no `pytest-timeout`,
so the CI job timeout is the backstop.
## Postgres
No equivalent change needed. It pages the whole thread with no id bound and follows parent pointers
in Python, and its upsert never rewrites `parent_checkpoint_id`, so it can neither miss this parent
nor form the loop. `BaseCheckpointSaver` and `InMemorySaver` also walk parent pointers.
## Test plan
New `libs/checkpoint-sqlite/tests/test_delta_parent_walk.py`:
- [x] Sync and async, parametrised over both id orders; the sync case also asserts equality with
`BaseCheckpointSaver` on the same rows. `parent_id_sorts_above_child` is the bug,
`parent_id_sorts_below_child` the control.
- [x] `test_walk_reaches_root_of_long_chain_with_descending_ids`: 40 checkpoints, only stored value
at the root.
- [x] `test_walk_terminates_when_put_makes_the_parent_chain_cycle`.
- [x] `test_walk_step_looks_up_the_parent_by_primary_key`: asserts the recursive step's
`EXPLAIN QUERY PLAN` is a key lookup, so a plain `JOIN` can't come back. Fails with it.
- [x] On `main`: 3 of the 6 walk tests fail (both `parent_id_sorts_above_child` cases and the long
chain). The cycle test passes on `main` too, since the old bounded scan could not loop; it
guards the new path.
- [x] #8550's repro returns `{'writes': [('task', 'ch', 'write-root')], 'seed': 'seed'}` sync and
async (was `{'writes': []}` on `main`).
- [x] `libs/checkpoint-sqlite`: `make format`, `make lint` clean; full suite 125 passed, 2 skipped.
- [x] `libs/langgraph`: `-k "delta or sqlite"` 739 passed, 1 skipped.
Thanks to @lylelllll for the report, the minimal repro, the base-saver comparison that isolated it
to the fast path, and for suggesting the recursive CTE.
Co-authored-by: lylelllll <59271327+lylelllll@users.noreply.github.com>
Follow-up to review on #8540, where a stale `# noqa: E402` slipped past
me and Sydney spotted it by eye. This turns on the rule that catches
that automatically.
`RUF100` flags a `noqa` that suppresses nothing. `sdk-py` already had it
through its blanket `RUF` selection; this adds it to the other seven
packages and clears what it finds.
### The 33 it flags, all autofixed
**Blanket `# noqa` on docstring-closing lines** (4, in
`checkpoint-postgres` and `checkpoint-sqlite`). `E501` is in
`lint.ignore` for those packages, so nothing was being suppressed:
```diff
- """ # noqa
+ """
```
**`# noqa: F821` on `anext(aiter_)`** (2). Left over from Python 3.9
support. `anext` became a builtin in 3.10, which is the floor now, so
`F821` no longer fires:
```diff
- anext(aiter_), # type: ignore[arg-type] # noqa: F821
+ anext(aiter_), # type: ignore[arg-type]
```
**Suppressions naming rules the package does not enable** (27), across
`langgraph`, `prebuilt` and `checkpoint-sqlite`: `FBT001`, `FBT002`,
`TC002`, `BLE001`, `ANN001`, `ANN002`, `ANN003`, `E501`, `F401`. Mostly
copied between packages whose rule sets differ.
### One measurement note
If you check these numbers yourself, use `--extend-select`:
```
ruff check --select RUF100 . # 81, misleading
ruff check --extend-select RUF100 . # 33, real
```
With a bare `--select`, ruff treats every other rule as disabled, so
every suppression for another rule looks unused. I quoted 81 before
catching that.
### Verified
`checkpoint-sqlite` 118 passed, `prebuilt` 284 passed, `langgraph` 1968
passed, `checkpoint-postgres` 264 passed on PG 15 and 16. `make lint`
clean in every package.
Independent of #8540 and #8537, so it can land in any order.
## Summary
Adds a sqlite-specific override of
`BaseCheckpointSaver.get_delta_channel_history` (and async). Before this
PR, `SqliteSaver` / `AsyncSqliteSaver` inherited the default impl, which
calls `get_tuple` once per ancestor — N round-trips, full pending-writes
fetch per step regardless of channel relevance.
The override mirrors the postgres two-stage shape (ancestor walk +
per-channel UNION ALL writes fetch) but adapted for sqlite:
- **No JSONB** → stage 1 streams the cursor row-by-row in
`checkpoint_id` DESC order. The merged walk advances one row at a time,
deserializing only on-path checkpoints and dropping each before
advancing — peak in-flight is one deserialized checkpoint, no
`fetchall()` materialization.
- **No separate blob table** → `channel_values` lives inline in the
checkpoint blob, so seeds come back from stage 1 with no second fetch.
- **Single merged walk (not K independent walks)**: each visited cid is
deserialized exactly once, regardless of how many channels are still
seeking their seed.
- **Stage 2** stays per-channel UNION ALL to avoid over-fetching writes
when channels have different chain depths — same rationale as postgres.
`AsyncSqliteSaver.get_delta_channel_history` bridges to its async form
via `run_coroutine_threadsafe`, matching the same cross-thread guard
used by `get_tuple` / `delete_thread`.
## Tests
- New `tests/test_delta_channel_migration.py`: covers the
`BinaryOperatorAggregate -> DeltaChannel` migration path on sqlite (sync
round-trip, sync continuation with post-migration delta folding, async
round-trip). Mirrors
`libs/langgraph/tests/test_delta_channel_migration.py` (which covered
`InMemorySaver`); without these, the override's behavior on
pre-migration threads was unverified — the override has to identify a
plain accumulated `channel_values[ch]` at a pre-migration ancestor as a
valid `seed`, not just `_DeltaSnapshot` sentinels.
- Existing `tests/test_get_delta_channel_history.py` (7 tests) continues
to pass and now exercises the optimized override end-to-end (previously
hit the inherited default impl).
- `make format`, `make lint`, `make test`: clean. 97/97 in the non-flaky
sqlite suite (the one ignored test, `test_async_asearch_refresh_ttl`, is
a known TTL-store timing flake on a separate module unrelated to this
PR).
## Benchmarks
### `get_delta_channel_history` micro-bench (override vs inherited
default impl)
1000-turn synthetic threads with sentinel snapshots + per-step writes;
`bench_sqlite_delta_history.py`. Per-call latency in microseconds.
| Scenario | min | median | mean |
|---|---:|---:|---:|
| S1 single channel, root-only snapshot | **4.60x** | **4.90x** |
**5.13x** |
| S2 mixed cadence (every-50 + root-only), 2 channels | **6.08x** |
**6.37x** | **6.84x** |
| S3 K=8 channels, root-only snapshot | 1.23x | 1.27x | 0.90x |
S2 wins biggest because per-channel UNION ALL avoids over-fetching
writes for the shallow channel. S3 is the worst case for sqlite (8
channels all walking to root, 1000 deserializations either way) — the
override still wins on min/median.
### Long-running thread mem/storage bench (delta vs no-delta)
`bench_sqlite_delta_memory.py`. `delta` mode uses `DeltaChannel` + the
override; `no_delta` uses `Annotated[list, _messages_delta_reducer]`
(full state in every blob). Same workload, file-backed sqlite. Latency
measured untraced (30 iterations); peak heap measured separately under
tracemalloc.
| Scenario | Turns | Storage Δ | Peak heap Δ | Read latency Δ |
|---|---:|---|---|---|
| K=1, freq=50 | 200 | **-96%** (942 KB vs 25.1 MB) | +21% (504 KB vs
418 KB) | **+13%** |
| K=1, freq=50 | 500 | **-98%** (2.9 MB vs 152.3 MB) | +20% (1.2 MB vs
1.0 MB) | **-6%** (delta wins) |
| K=3, freq=50 uniform | 200 | **-98%** (1.7 MB vs 73.5 MB) | +7% (1.3
MB vs 1.2 MB) | **+10%** |
| K=3, freq=50 uniform | 500 | **-99%** (6.0 MB vs 452.5 MB) | +7% (3.3
MB vs 3.0 MB) | **+6%** |
| K=3, freq=mixed | 200 | **-98%** (1.4 MB vs 73.5 MB) | +5% (1.3 MB vs
1.2 MB) | +190% (5.1 ms vs 1.7 ms abs) |
| K=3, freq=mixed | 500 | **-99%** (4.1 MB vs 452.5 MB) | +8% (3.3 MB vs
3.0 MB) | +377% (20.9 ms vs 4.4 ms abs) |
- **Storage**: -96 to -99% on long threads (a 500-turn K=3 thread
shrinks from 452 MB to 6 MB on disk). This is the headline win.
- **Peak heap**: within +5 to +21% of the no-delta path — the streaming
cursor + merged walk + drop-after-deserialize keep peak in-flight at one
checkpoint at a time.
- **Read latency**: equivalent-ish (within ~15%) on uniform-cadence
scenarios; at K=1/500 turns delta even wins by 6%. The mixed-cadence
rows have one channel with `snapshot_frequency=1000` walking to root on
a 500-turn thread — by configuration. Absolute mixed-delta latency is
still 5-21 ms per read.
Bench scripts (not committed; workspace-root convention matches other
`bench_*.py` files):
- `bench_sqlite_delta_history.py`
- `bench_sqlite_delta_memory.py`
## Test plan
- [x] `cd libs/checkpoint-sqlite && make format` clean
- [x] `cd libs/checkpoint-sqlite && make lint` clean
- [x] `cd libs/checkpoint-sqlite && make test` — 97 passed (1 known
flake unrelated)
- [x] `tests/test_get_delta_channel_history.py` — 7/7 (now exercises the
override)
- [x] `tests/test_delta_channel_migration.py` — 3/3 (new)
`aiosqlite` changed it's Connection type to no longer subclass
`threading.Thread`. This removed the is_alive method, which is called
proactively in setup().
This PR handles this in a backwards compat way.
This PR updates the dependencies in all Python packages using `uv lock
--upgrade`.
This is an automated PR created by the UV Lock Upgrade workflow.
To make tests pass:
* linting fixes
* whitespace fixes in snapshots
---------
Co-authored-by: sydney-runkle <54324534+sydney-runkle@users.noreply.github.com>
Co-authored-by: Sydney Runkle <sydneymarierunkle@gmail.com>
Prepare langgraph-checkpoint for 0.5
- Given we have no upper bound on langgraph-checkpoint dep need to undo all changes in langgraph-checkpoint that might break previous versions of langgraph
- Instead store sends in a Topic channel, removing the need to fetch sends as writes against the parent checkpoint
- Remove deprecated/unused functions in langgraph-checkpoint (will require bumping min range for langgraph-checkpoint in langgraph lib)
- Implement migration of old pending sends in langgraph-checkpoint-postgres
- Ensure parent config of `checkpoint_during=False` checkpoints always points to checkpoints that were also saved
* Migrate to `uv`
* Format `pyproject.toml` files properly
* Remove upper bounds on dependencies, and bounds on dev dependencies
(we should be using latest)
* Move to hatch for packaing
In the future we should:
* Set up dependabot / automate lockfile updates and tests
* Add tests for min compatible versions (I'll do this right after merge)
* Use dynamic versioning
* Bump `pydantic` to v2.11.4 in the lockfile, we have some tests failing
- Deletes all data associated with a thread_id
- Implemented in InMemory, Sqlite and Postgres checkpointers
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
**Description:**
Make AsyncSqliteSaver examples workable.
**Issue:**
For "Usage within StateGraph" example,
SyntaxError: 'async with' outside async function
For "Raw usage" example
KeyError: 'checkpoint_ns' and KeyError: 'id'
**Dependencies:**
N/A
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
Replace hardcoded database saver class names with `cls` in
`from_conn_string` factory methods to improve subclassing support
## Changes
* Replaced direct class instantiations with `cls(conn)` in
`from_conn_string` classmethods across all database implementations
* Updated both synchronous and asynchronous variants for DuckDB,
PostgreSQL, and SQLite savers
## Why
This refactor makes the database saver classes more extensible by
following Python's convention of using `cls` in class methods. This
enables proper inheritance patterns where subclasses can reuse the
factory methods without needing to override them. Previously, the
hardcoded class names would always instantiate the parent class, even
when called from a subclass.
## Testing
The change is backward compatible and doesn't alter existing
functionality. All existing tests should continue to pass as this is
purely a structural refactoring that preserves the current behavior
while improving extensibility.
## Notes
This PR addresses follow up on comments from #2518 - AsyncPostgresSaver
didn't need to be fixed but many of the other DB saver classes did.
* Performance improvements in checkpointer libs
- Use sha1 instead of md5 for hashing (faster in python 3.x)
- Use orjson instead of json for json dumping (sadly can't use for json loading)
* Update tests
* Update
* Use random number instead of hash for get_version_number
* Avoid saving writes for the last task to complete in each step
- only when possible, exceptions for ERROR, INTERRUPT, SEND
* Make Channel.from_checkpoint a regular function
- context manager no longer needed since Context became a managed value
* Use __slots__ for Channels
* Fix for kafka
* Fix semantics of put_writes/list
- put_writes(error) should not prevent saving future successful if task is retried successfully
- put_writes(writes) should be a no-op if non-error writes already exist for that task (this prevents tasks executed more than once from modifying writes previously saved / acted on)
- checkpoints should not include channel default values (ie those without a version)
- list() should fetch and return writes for each checkpoint
* Lint
* Rm print
* Fix import
* Lint
