Commit Graph
5 Commits
Author SHA1 Message Date
Nuno Campos bdabb723b0 Add langgraph dep 2024-09-10 16:39:34 -07:00
Nuno Campos 98b8595a94 Move to dev deps 2024-09-10 16:17:22 -07:00
Nuno Campos 5b3bd920d8 Catch errors thrown by nodes run by Executor 2024-09-10 16:17:22 -07:00
Nuno Campos e6d6f565a5 Improvements to consumer and producer patterns
- await future returned by send() instead of flush()
- use consumer groups by default
- process tasks in batches by default, configurable
- manually commit offsets when batch is processed
2024-09-10 16:17:22 -07:00
Nuno Campos 841aabf5b3 Implement LangGraph Scheduler for Kafka
- Orchestrator and Executor classes to run LangGraph in a distributed fashion using Kafka as a message bus for communication
- Orchestrator and Executor run on-demand when a new message is published to the topic they listen to
- Orchestrator is responsible for running the Pregel algorithm (deciding next tasks to run) and sending messages to the executor topic
- Executor is responsible for executing each task (node), and sending messages to the orchestrator topic when done
2024-09-10 16:17:22 -07:00