Previously, we validated that nodes explicitly route to END.
This feels a bit unnecessary, since graphs are expected to keep processing until no work is left to be done
Plus fix a small bug in validation.
- This introduces a new optional method BaseChannel.consume, which gets called when a channel triggers a node
- This is an alternative place to clean up channel state, in addition to the existing pattern of clearing state in the update() call for the next step
- Channels should implement consume() when they want to clean up state exactly if and only if the channel triggered a node
- Channels should clean up state in update() when they instead need to guarantee their value is available for a single step, irrespective of whether it was actually read
- any invalid utf-8 chars now removed on dumps
- fix serialization of Send
- remove serialization of NamedTuple, which doesn't work
- add test for custom serde passed to memory saver
- add test using Send and JsonPlus serde
- currently output of all nodes is streamed only when all nodes in that step finish
- checkpoint/values stream chunks still yielded only at the end of the step, as they require applying all updates
- reorganize checkpointing code to share code between input and step checkpoints
- do not emit debug/checkpoint events when there is no checkpointer attached
- emit debug/checkpoint event for input checkpoint as well
- current Pregel primitive is pull-based, ie. nodes write to channels, and it's up to other nodes to subscribe to those channels to "pull" updates
- this adds a "push" primitive where a node can directly schedule a node (more than once if desired) for execution in the next step, with additional kwargs to be passed in. Nodes scheduled in this way get called with both the current state and any kwargs passed to Packet
* Merge configurable fields with previous checkpoint config.
* Update update_state() and aupdate_state() to merge configurable fields with previous checkpoint config.
* Update tests to verify that all checkpoint metadata contain the expected configurable field keys. This assertion is needed because a run can have an arbitrary number of steps based on the construction of the graph.
Nuno Campos