Commit Graph
10 Commits
Author SHA1 Message Date
Nuno Campos d22fe60d4c serde: Handle unicode issues when encoding
- any invalid utf-8 chars now removed on dumps
- fix serialization of Send
- remove serialization of NamedTuple, which doesn't work
- add test for custom serde passed to memory saver
- add test using Send and JsonPlus serde
2024-06-03 16:28:32 -07:00
Andrew NguonlyandGitHub 08b505e017 Merge configurable fields with previous checkpoint config before each run (#510)
* Merge configurable fields with previous checkpoint config.

* Update update_state() and aupdate_state() to merge configurable fields with previous checkpoint config.

* Update tests to verify that all checkpoint metadata contain the expected configurable field keys. This assertion is needed because a run can have an arbitrary number of steps based on the construction of the graph.
2024-05-21 11:03:25 -07:00
Nuno Campos 52b2c755d7 Use uuid6 as the id for checkpoints
- this avoids conflicts if multiple processes creating checkpoints in same thread at same time
- uuid6 with a monotically increasing clock_seq is sortable by creation time at ms precision (plus clock_seq for ties, plus 48 bits of randomness for further ties)
2024-05-14 16:56:25 -07:00
Nuno Campos f160f82912 Add checkpoint metadata fields
- source: input, update or loop
- step: int
- make step counter continue from previous last step
2024-05-06 11:52:49 -07:00
Nuno Campos 3ff3def62b Remove option to only checkpoint at end of run
- Now that checkpoint at end of each step adds no latency there is not point to keep this
- This will make it easier to add future features
2024-05-06 11:52:49 -07:00
Nuno Campos f5a7c01e88 py39 2024-04-12 11:55:20 -07:00
Nuno Campos 74a40a895a Make ser/de configurable in checkpointer classes
- Remove pydantic usage from base checkpointer class
- Make serialization configurable for all existing checkpointer classes, you can eg use dill or json instead of pickle
2024-04-12 11:49:29 -07:00
Nuno Campos 3bfac1f490 Track parent relationships for checkpoints
- This enables building "branching" views of checkpoint history
2024-03-20 12:08:22 -07:00
Nuno Campos 0d13c6b159 Add history tracking to in-memory, sqlite and aiosqlite checkpointers
- Add Pregel.get_state_history and .aget_state_history methods to get history iterator
- Update checkpointer base class with new list and get_tuple methods
- Rewrite in-memory checkpointer class to track history
- Rewrite sqlite and aiosqlite checkpointers to track history
- Add new tests for history tracking
2024-03-13 17:22:39 -07:00
Nuno Campos 5c2afc9418 Fix behaviour of interrupt_before
- checkpoints are now copied before being mutated
- compiled graph and compiled state graph no longer need to override get_state/update_state
- pregel class now natively supports interrupt before/after
2024-02-24 19:43:30 -08:00