- The result of these doesnt change once a node is created, and it's
fairly expensive to run, so great thing to cache
- There's a variety of errors that can come from inspecting the source
code of a function (part of what this does) so adding a catch-all
try-except block as this should be best-effort, not crash your graph
- The result of these doesnt change once a node is created, and it's fairly expensive to run, so great thing to cache
- There's a variety of errors that can come from inspecting the source code of a function (part of what this does) so adding a catch-all try-except block as this should be best-effort, not crash your graph
- async tests are placed in test_pregel_async, not in test_pregel
- to avoid tests placed in wrong file being accidentally skipped i've
added the auto-async mark to sync test file
- this was not possible in async where all done callbacks are called in
next tick
- in sync case this would manifest as the first task done callback
seeing counter == 1 and thus setting event
- the fix is to unset the event whenever a task is scheduled
- When using an async entrypoint you can now freely mix and match sync
and async tasks with a uniform api (ie all tasks return a sync or async
future depending on context)
- Fix issues with scheduling deeply nested tasks (use threadsafe methods
to schedule coroutines and create futures)
- async tests are placed in test_pregel_async, not in test_pregel
- to avoid tests placed in wrong file being accidentally skipped i've added the auto-async mark to sync test file
- this was not possible in async where all done callbacks are called in next tick
- in sync case this would manifest as the first task done callback seeing counter == 1 and thus setting event
- the fix is to unset the event whenever a task is scheduled
- When using an async entrypoint you can now freely mix and match sync and async tasks with a uniform api (ie all tasks return a sync or async future depending on context)
- Fix issues with scheduling deeply nested tasks (use threadsafe methods to schedule coroutines and create futures)
- both issues are related to the fact that waiters for futures are
notified of completion before "done" callbacks are called
- 1st issue manifested as interrupt stream event being emitted before
the result of a task that logically finished first (it's in the line
above in body of the entrypoint function) -> this is solved by always
returning to use code a fresh future chained on the original future,
because chaining is done via done callbacks (therefore the chained
future will only resolve after done callbacks of the original feature
are called)
- 2nd issue mainfested as sometimes (very rarely) the last stream event
not being printed before stream() finishes. this is solved by ensuring
we only return out of PregelRunner.tick() once all "done" callbacks are
called, previously we were approximating this through use of
asyncio.sleep(0) / time.sleep(0). The new solution instead waits on a
threading/asyncio.Event which will only be set by the last "done"
callback to fire
- this PR also disables incomplete support for calling sync tasks from
async entrypoints
- both issues are related to the fact that waiters for futures are notified of completion before "done" callbacks are called
- 1st issue manifested as interrupt stream event being emitted before the result of a task that logically finished first (it's in the line above in body of the entrypoint function) -> this is solved by always returning to use code a fresh future chained on the original future, because chaining is done via done callbacks (therefore the chained future will only resolve after done callbacks of the original feature are called)
- 2nd issue mainfested as sometimes (very rarely) the last stream event not being printed before stream() finishes. this is solved by ensuring we only return out of PregelRunner.tick() once all "done" callbacks are called, previously we were approximating this through use of asyncio.sleep(0) / time.sleep(0). The new solution instead waits on a threading/asyncio.Event which will only be set by the last "done" callback to fire
- order was incorrectly based on task id, instead of the correct task
path
- this requires storing task paths on checkpointers
- addition of task_path to put_writes is made backwards compatible by
checking signature on call, and treating it as an optional arg
- order was incorrectly based on task id, instead of the correct task path
- this requires storing task paths on checkpointers
- addition of task_path to put_writes is made backwards compatible by checking signature on call, and treating it as an optinal arg
