txn-wal: Port txns_progress_source off the async operator builder#37127
Merged
Conversation
b53e4e1 to
5061e53
Compare
aljoscha
reviewed
Jun 18, 2026
txns_progress_source was the last operator in txn-wal still built on the async Timely operator builder. Its awaits are real persist I/O (open_writer, data_subscribe, rx.recv), so this ports it to a synchronous build_reschedule operator paired with a Tokio task that owns the persist resources and the subscription receiver. The task forwards DataRemapEntry values over an unbounded channel and wakes the operator through an ArcActivator. The physical_upper emit decision and the capability downgrades stay on the Timely thread, and a Finished sentinel lets the operator distinguish a clean subscription close from a task panic. Shutdown reproduces builder_async's two-phase handshake via button(), matching the already synchronous txns_progress_frontiers. No behavior change is intended. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5061e53 to
50c7e2b
Compare
aljoscha
approved these changes
Jun 19, 2026
aljoscha
left a comment
Contributor
There was a problem hiding this comment.
I think this is correct 🤞 it's also much simpler than the other operators you de-asyncified.
Member
Author
|
Thanks for the review! |
antiguru
added a commit
to antiguru/materialize
that referenced
this pull request
Jun 19, 2026
…orker shutdown Lessons from porting txns_progress_source (PR MaterializeInc#37127): - sync new_output takes the container type, not the container builder - spawning task work needs Send bounds the async builder did not - the let _ = &handle trick is a no-op when captured state owns the handle - single-worker-output operators still need every worker in the shutdown handshake Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
txns_progress_sourcewas the last operator in txn-wal still built on the async Timely operator builder.Its sibling
txns_progress_frontiersis already synchronous, so this removes the remaining async operator from the crate.Description
The operator's awaits are real persist I/O (
open_writer,data_subscribe,rx.recv), so it ports to a synchronousbuild_rescheduleoperator paired with a Tokio task that owns the persist resources and the subscription receiver.The task forwards
DataRemapEntryvalues over an unbounded channel and wakes the operator through anArcActivator.The
physical_upperemit decision and the capability downgrades stay on the Timely thread, so the operator still emits only when the physical upper advances and downgrades tological_upper, on a single chosen worker.A
Finishedsentinel lets the operator distinguish a clean subscription close from a task panic.Shutdown reproduces
builder_async's two-phase handshake viabutton()andlocal_pressed/all_pressed, matchingtxns_progress_frontiers.No behavior change is intended.
Verification
Covered by the existing in-memory persist integration tests that drive
txns_progressend-to-end (operator::tests::data_subscribe,as_of_until,subscribe_shard_finalize,subscribe_shard_register_forget).