Commit Graph
3338 Commits
Author SHA1 Message Date
Nuno Campos 571623442a Avoid closing drain before seeing first message in either topic 2024-09-11 16:45:43 -07:00
Nuno CamposandGitHub c172362918 Merge pull request #1691 from langchain-ai/nc/11sep/kafka-sync
kafka: Implement sync orchestrator and executor classes
2024-09-11 16:34:01 -07:00
Nuno CamposandGitHub 115c37acea Merge pull request #1686 from langchain-ai/nc/11sep/kafka-configurable-consumer-producer
kafka: Make consumer and producer classes configurable
2024-09-11 16:33:15 -07:00
Nuno Campos dd68babac7 kafka: Expose finally_send as public api
- Enables attaching a list of messages (topic/key/value) to send when run finishes
2024-09-11 16:31:35 -07:00
Vadym BardaandGitHub 63c34ca992 langgraph: update snapshots for 0.3.x core tests (#1692) 2024-09-11 19:28:47 -04:00
Nuno Campos 4accc19734 kafka: Implement sync orchestrator and executor classes 2024-09-11 16:06:50 -07:00
Vadym BardaandGitHub 84643b2789 langgraph: update core & snapshots (#1689) 2024-09-11 21:59:12 +00:00
Nuno Campos 217b07ae65 kafka: Make consumer and producer classes configurable
- Define protocol for sync and async producer and consumer
- Accept consumer/producer as init args in Orchestrator/Executor
- If not passed in, create default consumer/producer as before
2024-09-11 14:27:08 -07:00
Nuno Campos f179da00a0 update bench job 2024-09-11 13:41:45 -07:00
Nuno CamposandGitHub 8e275fa485 Merge pull request #1609 from langchain-ai/nc/4sep/benchmark
Add benchmark script
2024-09-11 12:29:22 -07:00
Nuno Campos 47dae71841 Remove files 2024-09-11 12:28:38 -07:00
Nuno Campos bf4f8e67d8 Rename command 2024-09-11 12:25:22 -07:00
Nuno Campos d844359d61 Fix output dir 2024-09-11 12:23:59 -07:00
Nuno Campos f577529356 Add benchmark/profile commands 2024-09-11 11:44:18 -07:00
Tat Dat Duong a60a4606fb fix(sdk-js): bump to 0.0.9 2024-09-11 19:41:51 +02:00
gbaian10andGitHub bf6ad19963 refactor: Type narrowing for ToolNode.name (#1678)
Avoid type incompatibility when using `add_node` to add a `ToolNode`.
2024-09-11 13:11:39 -04:00
Vadym BardaandGitHub f3b5d804ab ci: fix kafka scheduler, split langgraph and kafka into separate test jobs (#1677) 2024-09-11 10:55:11 -04:00
Nuno Campos 26f23ea26c Add benchmark script 2024-09-10 17:32:47 -07:00
Nuno Campos 5de46f7472 Update stop condition for drain 2024-09-10 16:52:09 -07:00
Nuno Campos bdabb723b0 Add langgraph dep 2024-09-10 16:39:34 -07:00
Nuno Campos edc0fd0210 Remove extraneous asserts 2024-09-10 16:20:45 -07:00
Nuno Campos 3886b76771 Lint 2024-09-10 16:17:22 -07:00
Nuno Campos 3a65460767 Use anyio for timeout in dran test helper 2024-09-10 16:17:22 -07:00
Nuno Campos 5e0273886e Lint 2024-09-10 16:17:22 -07:00
Nuno Campos 943cc4d83a Add readme, lint 2024-09-10 16:17:22 -07:00
Nuno Campos 270559b880 Implement changes in sqlite checkpointer 2024-09-10 16:17:22 -07:00
Nuno Campos 98b8595a94 Move to dev deps 2024-09-10 16:17:22 -07:00
Nuno Campos f7dffa023c Implement subgraph delegation for distributed arch 2024-09-10 16:17:22 -07:00
Nuno Campos 7dc69a543d Rename 2024-09-10 16:17:22 -07:00
Nuno Campos 8f91402642 Use thread_id for partition key, ignore tasks for stale checkpoints 2024-09-10 16:17:22 -07:00
Nuno Campos 030d6d2def Fix subgraph test 2024-09-10 16:17:22 -07:00
Nuno Campos ad7dd0dbb7 Add drain_topics test helper 2024-09-10 16:17:22 -07:00
Nuno Campos 0eb11a3cf9 Remove is_resuming arg 2024-09-10 16:17:22 -07:00
Nuno Campos 1ce8d628f2 In update state skip nodes not found in instance
eg, because removed since
2024-09-10 16:17:22 -07:00
Nuno Campos 5e6b58f847 Remove double shield usage
- we already shield from cancellation in PregelLoop so shouldnt be repeating in AsyncBackgroundExecutor
2024-09-10 16:17:22 -07:00
Nuno Campos b13db42568 WIP Subgraphs 2024-09-10 16:17:22 -07:00
Nuno Campos 83d36761a4 Rename compose projects 2024-09-10 16:17:22 -07:00
Nuno Campos d206621230 Rename test 2024-09-10 16:17:22 -07:00
Nuno Campos 8e6caab78c Add test for interrupts 2024-09-10 16:17:22 -07:00
Nuno Campos ad463ec90a Add more test assertions 2024-09-10 16:17:22 -07:00
Nuno Campos 2d02e5e0c5 Add error topic, add retries (eg for checkpointer database exceptions) 2024-09-10 16:17:22 -07:00
Nuno Campos 5b3bd920d8 Catch errors thrown by nodes run by Executor 2024-09-10 16:17:22 -07:00
Nuno Campos f09b130837 Use pool for postgres checkpointer 2024-09-10 16:17:22 -07:00
Nuno Campos e6d6f565a5 Improvements to consumer and producer patterns
- await future returned by send() instead of flush()
- use consumer groups by default
- process tasks in batches by default, configurable
- manually commit offsets when batch is processed
2024-09-10 16:17:22 -07:00
Nuno Campos dc5be3fd58 Rename 2024-09-10 16:17:22 -07:00
Nuno Campos 381ea9cec8 Rename 2024-09-10 16:17:22 -07:00
Nuno Campos 841aabf5b3 Implement LangGraph Scheduler for Kafka
- Orchestrator and Executor classes to run LangGraph in a distributed fashion using Kafka as a message bus for communication
- Orchestrator and Executor run on-demand when a new message is published to the topic they listen to
- Orchestrator is responsible for running the Pregel algorithm (deciding next tasks to run) and sending messages to the executor topic
- Executor is responsible for executing each task (node), and sending messages to the orchestrator topic when done
2024-09-10 16:17:22 -07:00
jessicaouandGitHub acd8325402 adding langchain academy link (#1643) 2024-09-10 12:04:19 -04:00
Tat Dat Duong 58c34cf40c fix(js): don't be strict on JS types for custom fetch 2024-09-09 17:48:55 +02:00
Vadym BardaandGitHub 282562f182 langgraph: add tests for MessagesState/AnyMessage (#1637)
* langgraph: add tests for MessagesState/AnyMessage

* bump to 0.3.0.dev4

* update

* update snapshots
2024-09-07 17:00:41 -04:00