coord: add temp workaround for noisy log message#8053
Merged
Conversation
We're currently getting the following log message at every restart where the catalog
contains a previously created persistent user table:
2021-08-21T00:45:08.608002Z ERROR coord::coord: failed to compact persisted tables: invalid compaction less than or equal to trace since Antichain { elements: [123456789] }: Antichain { elements: [0] }
This happens because the very first thing the coordinator does is inform tables
and sources that it is safe to compact up to the zero since frontier, which is
fine without persistence because on restart those things start up with the
zero since frontier anyway. Persistent tables piggyback off of the since updates
for the table's primary index, but unlike the primary index, they restart with
potentially a nonzero since frontier, which is what triggers the error above.
It's also currently unexplained why this only happens for persisted user tables and
not persistent system tables.
This temporary workaround was first introduced in MaterializeInc#7984, and then removed in MaterializeInc#7977
because we incorrectly thought the only time this error triggered was on create time,
when we get a since update at t = 0 for a newly created persistent table that already
has a since frontier of 0.
Contributor
Author
|
edited the commit message and the comment in the code to be more specific that the issue is on restart, otherwise this is the same patch as #7984 |
danhhz
approved these changes
Aug 25, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We're currently getting the following log message at every restart where the catalog
contains a previously created persistent user table:
2021-08-21T00:45:08.608002Z ERROR coord::coord: failed to compact persisted tables: invalid compaction less than or equal to trace since Antichain { elements: [123456789] }: Antichain { elements: [0] }
This happens because the very first thing the coordinator does is inform tables
and sources that it is safe to compact up to the zero since frontier, which is
fine without persistence because on restart those things start up with the
zero since frontier anyway. Persistent tables piggyback off of the since updates
for the table's primary index, but unlike the primary index, they restart with
potentially a nonzero since frontier, which is what triggers the error above.
It's also currently unexplained why this only happens for persisted user tables and
not persistent system tables.
This temporary workaround was first introduced in #7984, and then removed in #7977
because we incorrectly thought the only time this error triggered was on create time,
when we get a since update at t = 0 for a newly created persistent table that already
has a since frontier of 0.