Commit Graph
15 Commits
Author SHA1 Message Date
Vadym BardaandGitHub d32386f849 checkpoint: add DuckDB checkpointer (#2145) 2024-10-23 21:11:03 +00:00
Vadym BardaandGitHub 03be3b6567 docs: reorganize API refs (#1884) 2024-09-27 18:12:12 -04:00
Nuno Campos 6c0339ca4b ci: Enable mypy checks for checkpoint-sqlite lib 2024-09-19 08:46:44 -07:00
Nuno Campos efb572f15b Add missing branch 2024-09-18 13:18:37 -07:00
Nuno CamposandGitHub 66fc7c96aa More performance improvements in checkpointing and channels (#1685)
* Performance improvements in checkpointer libs

- Use sha1 instead of md5 for hashing (faster in python 3.x)
- Use orjson instead of json for json dumping (sadly can't use for json loading)

* Update tests

* Update

* Use random number instead of hash for get_version_number

* Avoid saving writes for the last task to complete in each step

- only when possible, exceptions for ERROR, INTERRUPT, SEND

* Make Channel.from_checkpoint a regular function

- context manager no longer needed since Context became a managed value

* Use __slots__ for Channels

* Fix for kafka
2024-09-12 21:08:30 -07:00
Nuno Campos 270559b880 Implement changes in sqlite checkpointer 2024-09-10 16:17:22 -07:00
Nuno CamposandNuno Campos 4793b3f5e1 Use a lock for all operations on sqlite checkpointer
- Otherwise when used in multiple subgraphs in parallel separate queries can interfere w each other
2024-09-03 09:20:17 -07:00
Vadym BardaandGitHub d492475c61 checkpoint-sqlite: close connection in from_conn_string (#1589) 2024-09-03 11:00:37 -04:00
Nuno Campos e744db169a postgres/sqlite: Sort pending writes when fetching from db 2024-08-30 14:07:46 -07:00
Nuno CamposandGitHub a261e1a497 Fix semantics of put_writes/list (#1436)
* Fix semantics of put_writes/list

- put_writes(error) should not prevent saving future successful if task is retried successfully
- put_writes(writes) should be a no-op if non-error writes already exist for that task (this prevents tasks executed more than once from modifying writes previously saved / acted on)
- checkpoints should not include channel default values (ie those without a version)
- list() should fetch and return writes for each checkpoint

* Lint

* Rm print

* Fix import

* Lint
2024-08-22 19:04:42 +00:00
Nuno Campos 7aaeedd7ba checkpoint-*: In put_writes clear any previously saved writes for this task_id
- Previously implementations were clearing only tasks if the index matched a previously saved one, which isn't enough to guarantee idempotency
2024-08-19 15:40:04 -07:00
Vadym BardaandGitHub 71ce07d971 checkpoint: update docstrings for checkpoint libraries (#1277) 2024-08-08 16:26:16 +00:00
b37f78942d checkpoint-postgres: new library for postgres checkpointer implementation (#1236)
* checkpoint-postgres: new library for postgres checkpointer implementation

---------

Co-authored-by: Nuno Campos <nuno@langchain.dev>
2024-08-06 22:37:06 -04:00
vbarda 64c30508d9 checkpoint-sqlite: stop using checkpointers as context managers, use contextmanager only in from_conn_string 2024-08-05 16:20:16 -04:00
Vadym BardaandGitHub c149a99b44 checkpoint-sqlite: new library for sqlite checkpointer implementation (#1203)
* checkpoint-sqlite: new library for sqlite checkpointer implementation
2024-08-02 22:14:12 +00:00