Nuno Campos
571623442a
Avoid closing drain before seeing first message in either topic
2024-09-11 16:45:43 -07:00
Nuno Campos and GitHub
c172362918
Merge pull request #1691 from langchain-ai/nc/11sep/kafka-sync
...
kafka: Implement sync orchestrator and executor classes
2024-09-11 16:34:01 -07:00
Nuno Campos and GitHub
115c37acea
Merge pull request #1686 from langchain-ai/nc/11sep/kafka-configurable-consumer-producer
...
kafka: Make consumer and producer classes configurable
2024-09-11 16:33:15 -07:00
Nuno Campos
dd68babac7
kafka: Expose finally_send as public api
...
- Enables attaching a list of messages (topic/key/value) to send when run finishes
2024-09-11 16:31:35 -07:00
Vadym Barda and GitHub
63c34ca992
langgraph: update snapshots for 0.3.x core tests ( #1692 )
2024-09-11 19:28:47 -04:00
Nuno Campos
4accc19734
kafka: Implement sync orchestrator and executor classes
2024-09-11 16:06:50 -07:00
Vadym Barda and GitHub
84643b2789
langgraph: update core & snapshots ( #1689 )
2024-09-11 21:59:12 +00:00
Nuno Campos
217b07ae65
kafka: Make consumer and producer classes configurable
...
- Define protocol for sync and async producer and consumer
- Accept consumer/producer as init args in Orchestrator/Executor
- If not passed in, create default consumer/producer as before
2024-09-11 14:27:08 -07:00
Nuno Campos
f179da00a0
update bench job
2024-09-11 13:41:45 -07:00
Nuno Campos and GitHub
8e275fa485
Merge pull request #1609 from langchain-ai/nc/4sep/benchmark
...
Add benchmark script
2024-09-11 12:29:22 -07:00
Nuno Campos
47dae71841
Remove files
2024-09-11 12:28:38 -07:00
Nuno Campos
bf4f8e67d8
Rename command
2024-09-11 12:25:22 -07:00
Nuno Campos
d844359d61
Fix output dir
2024-09-11 12:23:59 -07:00
Nuno Campos
f577529356
Add benchmark/profile commands
2024-09-11 11:44:18 -07:00
Tat Dat Duong
a60a4606fb
fix(sdk-js): bump to 0.0.9
2024-09-11 19:41:51 +02:00
gbaian10 and GitHub
bf6ad19963
refactor: Type narrowing for ToolNode.name ( #1678 )
...
Avoid type incompatibility when using `add_node` to add a `ToolNode`.
2024-09-11 13:11:39 -04:00
Vadym Barda and GitHub
f3b5d804ab
ci: fix kafka scheduler, split langgraph and kafka into separate test jobs ( #1677 )
2024-09-11 10:55:11 -04:00
Nuno Campos
26f23ea26c
Add benchmark script
2024-09-10 17:32:47 -07:00
Nuno Campos
5de46f7472
Update stop condition for drain
2024-09-10 16:52:09 -07:00
Nuno Campos
bdabb723b0
Add langgraph dep
2024-09-10 16:39:34 -07:00
Nuno Campos
edc0fd0210
Remove extraneous asserts
2024-09-10 16:20:45 -07:00
Nuno Campos
3886b76771
Lint
2024-09-10 16:17:22 -07:00
Nuno Campos
3a65460767
Use anyio for timeout in dran test helper
2024-09-10 16:17:22 -07:00
Nuno Campos
5e0273886e
Lint
2024-09-10 16:17:22 -07:00
Nuno Campos
943cc4d83a
Add readme, lint
2024-09-10 16:17:22 -07:00
Nuno Campos
270559b880
Implement changes in sqlite checkpointer
2024-09-10 16:17:22 -07:00
Nuno Campos
98b8595a94
Move to dev deps
2024-09-10 16:17:22 -07:00
Nuno Campos
f7dffa023c
Implement subgraph delegation for distributed arch
2024-09-10 16:17:22 -07:00
Nuno Campos
7dc69a543d
Rename
2024-09-10 16:17:22 -07:00
Nuno Campos
8f91402642
Use thread_id for partition key, ignore tasks for stale checkpoints
2024-09-10 16:17:22 -07:00
Nuno Campos
030d6d2def
Fix subgraph test
2024-09-10 16:17:22 -07:00
Nuno Campos
ad7dd0dbb7
Add drain_topics test helper
2024-09-10 16:17:22 -07:00
Nuno Campos
0eb11a3cf9
Remove is_resuming arg
2024-09-10 16:17:22 -07:00
Nuno Campos
1ce8d628f2
In update state skip nodes not found in instance
...
eg, because removed since
2024-09-10 16:17:22 -07:00
Nuno Campos
5e6b58f847
Remove double shield usage
...
- we already shield from cancellation in PregelLoop so shouldnt be repeating in AsyncBackgroundExecutor
2024-09-10 16:17:22 -07:00
Nuno Campos
b13db42568
WIP Subgraphs
2024-09-10 16:17:22 -07:00
Nuno Campos
83d36761a4
Rename compose projects
2024-09-10 16:17:22 -07:00
Nuno Campos
d206621230
Rename test
2024-09-10 16:17:22 -07:00
Nuno Campos
8e6caab78c
Add test for interrupts
2024-09-10 16:17:22 -07:00
Nuno Campos
ad463ec90a
Add more test assertions
2024-09-10 16:17:22 -07:00
Nuno Campos
2d02e5e0c5
Add error topic, add retries (eg for checkpointer database exceptions)
2024-09-10 16:17:22 -07:00
Nuno Campos
5b3bd920d8
Catch errors thrown by nodes run by Executor
2024-09-10 16:17:22 -07:00
Nuno Campos
f09b130837
Use pool for postgres checkpointer
2024-09-10 16:17:22 -07:00
Nuno Campos
e6d6f565a5
Improvements to consumer and producer patterns
...
- await future returned by send() instead of flush()
- use consumer groups by default
- process tasks in batches by default, configurable
- manually commit offsets when batch is processed
2024-09-10 16:17:22 -07:00
Nuno Campos
dc5be3fd58
Rename
2024-09-10 16:17:22 -07:00
Nuno Campos
381ea9cec8
Rename
2024-09-10 16:17:22 -07:00
Nuno Campos
841aabf5b3
Implement LangGraph Scheduler for Kafka
...
- Orchestrator and Executor classes to run LangGraph in a distributed fashion using Kafka as a message bus for communication
- Orchestrator and Executor run on-demand when a new message is published to the topic they listen to
- Orchestrator is responsible for running the Pregel algorithm (deciding next tasks to run) and sending messages to the executor topic
- Executor is responsible for executing each task (node), and sending messages to the orchestrator topic when done
2024-09-10 16:17:22 -07:00
jessicaou and GitHub
acd8325402
adding langchain academy link ( #1643 )
2024-09-10 12:04:19 -04:00
Tat Dat Duong
58c34cf40c
fix(js): don't be strict on JS types for custom fetch
2024-09-09 17:48:55 +02:00
Vadym Barda and GitHub
282562f182
langgraph: add tests for MessagesState/AnyMessage ( #1637 )
...
* langgraph: add tests for MessagesState/AnyMessage
* bump to 0.3.0.dev4
* update
* update snapshots
2024-09-07 17:00:41 -04:00