mirror of
https://github.com/langchain-ai/langgraph.git
synced 2026-10-19 14:31:47 +02:00
cba111d8d6
A DeltaChannel is rebuilt from its writes along the parent chain, so a checkpoint saved without one of them reads back short for good. If saving a DeltaChannel write failed (a database blip, say), the next checkpoints were saved anyway and the channel lost that write, while plain channels kept it. Three things let that happen: - the sync loop waited for the delta writes before saving a checkpoint, but never checked that they succeeded; - both loops saved a checkpoint even when the previous save had failed, so it pointed at a parent that was never saved; - each save took the pending write list only when it started in the background, and saves can start out of order, so a save could take a later checkpoint's writes and not wait for its own. Waiting on writes queued behind it, a save could also hang a sync run with `durability="async"` (the default) once every worker was such a save: on a 15-core machine a loop of 100 fast steps hung every time with default settings, and with `max_concurrency=1` three steps are enough. Now `_put_checkpoint` hands each save the writes submitted before it, and a save fails if one of them or the previous save failed. The run raises the write's error as before, the thread stays at its last checkpoint that has all its writes, and running it again reruns the task. For graphs without a DeltaChannel, a failed save now also stops the later saves of that run, as JS already does, instead of saving checkpoints whose parent is missing. Thanks @shivangsharma01 for reproducing this in #8234, and @iroiro147, whose #8299 had the first two changes before the issue-assignment check closed it. Thanks also to @HarshShroff for the failed-save test case. JS twin: langchain-ai/langgraphjs#2969 ## Tests `test_delta_channel_write_failures.py`: - a saver fails one delta write once; on every durability, with `invoke` and `ainvoke`, every saved checkpoint matches a plain channel and a retry gets the write back. The `invoke` cases and `ainvoke` with `durability="async"` fail on main. - a saver fails the save of the checkpoint after `a` once; with `invoke` and `ainvoke`, every saved checkpoint matches a plain channel and a retry finishes the run. The `durability="async"` cases fail on main, which saves the later checkpoints on top of the missing one. - a delta graph run with `max_concurrency=1` finishes. It hangs on main. Reverting any one of the three changes fails at least one of these. `make format`, `make lint` and the langgraph suite pass. A 100-step benchmark (delta and plain channels, every durability, with and without 1 ms of saver latency) makes the same saver calls and runs in the same time as main, except for the case that hangs on main. --------- Co-authored-by: iroiro147 <265728356+iroiro147@users.noreply.github.com>