clear durability frontier on source drop#11945
Closed
guswynn wants to merge 1 commit into
Closed
Conversation
1 task
Contributor
Author
|
closing in favor at #11987, with possible adjustments |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
SourceToken's created byimport_source/create_sourceare ultimately held in thecapturetimely operator:materialize/src/storage/src/boundary/tcp_boundary.rs
Lines 301 to 311 in 98added
Apparently, for those to be dropped, the
durability_capabilityof thesourceoperator needs to be exhausted (advancedto&[]. This capability, as far as I can tell, is derived from thedatacapability of the source, and can lag behind.Curiously,
SimpleSourcesources always advance this cap, and are correctly dropped whenDROP SOURCEis used.SourceReadersources (the majority, including kafka), continue to advance the durability cap to the LASTdurability_frontierof the timestamp binding box"materialize/src/storage/src/source/mod.rs
Lines 969 to 970 in ff45f28
Motivation
Tips for reviewer
I could be misunderstanding some subtle ordering issue here, and its unclear to me if advancing the durability frontier causes the system to make some sort of misguided claim that "the source is finished, forever". If that is the case, then we will likely need to change the
durability_frontierapi to return anOption<Antichain<Timestamp>>instead, to indicate that the current state of the source is "dead". However, this may not matter, as dropped sources have id's who are never reused, so in some sense, the entirety of the source as we know it is durably persisted. One subtlety here is that sources can live and produce more data for a short time after they are dropped, as the system realizes they are dead (well, i may be wrong about this).Separately, @petrosagg
ts_source_mappinglooks like it is NEVER read, and should probably be removed?Testing
Release notes
This PR includes the following user-facing behavior changes: