Invalidate the StatefulExecutorCache on checkpoint restore - #745
Invalidate the StatefulExecutorCache on checkpoint restore#745PratikDhanave wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a StatefulExecutorCache hook that can be directly wired into an executor’s checkpoint-restore lifecycle so cached typed state is invalidated after an in-place checkpoint restore (preventing stale reads).
Changes:
- Introduce
StatefulExecutorCache.OnCheckpointRestored(*Context) error, which clears the cache by delegating toReset(). - Wire the new hook into the shared “aggregator” test binding via
Executor.OnCheckpointRestoredFunc. - Add an end-to-end test covering in-place checkpoint restore behavior to ensure the post-restore read observes restored state, not stale cached state.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
workflow/executor.go |
Adds a cache-invalidation hook compatible with Executor.OnCheckpointRestoredFunc to prevent stale cached state after restore. |
workflow/executor_test.go |
Wires the hook in a shared binding and adds a regression test validating correct behavior across checkpoint restore + resume. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This comment has been minimized.
This comment has been minimized.
StatefulExecutorCache keeps an executor-local value after its first read, and the only invalidation is Reset(), whose signature does not match the executor's OnCheckpointRestored hook. A shared stateful executor restored in place to an earlier checkpoint therefore kept its pre-restore cached value, and the next read returned the stale state instead of the imported checkpoint state. Add OnCheckpointRestored to StatefulExecutorCache. Its signature matches Executor.OnCheckpointRestoredFunc so it can be wired into the restore hook that notifyCheckpointLoaded already fires, clearing the cache so the next read observes the restored state. This mirrors the .NET/Python cache behavior, where in-memory executor state is dropped on checkpoint restore.
a92b1e5 to
648ba36
Compare
Cross-Repo Parity Review ✅PR summary: Adds ScopeThis PR exports one new method on Cross-repo parity assessmentThe Neither Python nor .NET exposes a direct Labels added: Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
What
StatefulExecutorCachecaches its typed state value after the first read (whenconcurrent runs are disabled and the caller does not skip the cache). The only
way to drop that cached value was
Reset(), whosefunc() errorsignature doesnot match the executor's
OnCheckpointRestoredFunchook.On an in-place checkpoint restore,
restoreCheckpointCoreimports the checkpointstate and then calls
notifyCheckpointLoaded, which fires each executor'sOnCheckpointRestored. A shared stateful executor that had already cached keptcached == trueholding the pre-restore value, so the nextReadStatereturnedthe stale value instead of the state imported from the checkpoint.
This adds
StatefulExecutorCache.OnCheckpointRestored(*Context) error, whichinvalidates the cache. Its signature matches
Executor.OnCheckpointRestoredFunc,so a shared stateful executor can wire it directly into the restore hook that the
runner already invokes.
Why
This matches the .NET/Python Agent Framework semantics, where in-memory executor
state derived from workflow state is dropped when a checkpoint is restored, so
the executor observes exactly the restored state on the next read. Restoring an
earlier checkpoint while still trusting a newer cached aggregate is a correctness
bug across SDKs.
How it is tested
TestStatefulExecutorCache_CheckpointRestoreInvalidatesCachebuilds a workflowfrom the shared
aggregatorBindingpattern, runsa(capturing the post-acheckpoint) then
bso the cache holdsa+b, restores in place to the post-acheckpoint, and sends
c. With the hook wired the aggregate is[a, a+b, a+c];without it the cache stays stale and yields
a+b+c. The existing cache andreset tests continue to pass.