Skip to content

coord: add temp workaround for noisy log message#8053

Merged
ruchirK merged 1 commit into
MaterializeInc:mainfrom
ruchirK:temp-log-fix
Aug 25, 2021
Merged

coord: add temp workaround for noisy log message#8053
ruchirK merged 1 commit into
MaterializeInc:mainfrom
ruchirK:temp-log-fix

Conversation

@ruchirK

@ruchirK ruchirK commented Aug 25, 2021

Copy link
Copy Markdown
Contributor

We're currently getting the following log message at every restart where the catalog
contains a previously created persistent user table:

2021-08-21T00:45:08.608002Z ERROR coord::coord: failed to compact persisted tables: invalid compaction less than or equal to trace since Antichain { elements: [123456789] }: Antichain { elements: [0] }

This happens because the very first thing the coordinator does is inform tables
and sources that it is safe to compact up to the zero since frontier, which is
fine without persistence because on restart those things start up with the
zero since frontier anyway. Persistent tables piggyback off of the since updates
for the table's primary index, but unlike the primary index, they restart with
potentially a nonzero since frontier, which is what triggers the error above.

It's also currently unexplained why this only happens for persisted user tables and
not persistent system tables.

This temporary workaround was first introduced in #7984, and then removed in #7977
because we incorrectly thought the only time this error triggered was on create time,
when we get a since update at t = 0 for a newly created persistent table that already
has a since frontier of 0.

We're currently getting the following log message at every restart where the catalog
contains a previously created persistent user table:

2021-08-21T00:45:08.608002Z ERROR coord::coord: failed to compact persisted tables: invalid compaction less than or equal to trace since Antichain { elements: [123456789] }: Antichain { elements: [0] }

This happens because the very first thing the coordinator does is inform tables
and sources that it is safe to compact up to the zero since frontier, which is
fine without persistence because on restart those things start up with the
zero since frontier anyway. Persistent tables piggyback off of the since updates
for the table's primary index, but unlike the primary index, they restart with
potentially a nonzero since frontier, which is what triggers the error above.

It's also currently unexplained why this only happens for persisted user tables and
not persistent system tables.

This temporary workaround was first introduced in MaterializeInc#7984, and then removed in MaterializeInc#7977
because we incorrectly thought the only time this error triggered was on create time,
when we get a since update at t = 0 for a newly created persistent table that already
has a since frontier of 0.
@ruchirK

ruchirK commented Aug 25, 2021

Copy link
Copy Markdown
Contributor Author

edited the commit message and the comment in the code to be more specific that the issue is on restart, otherwise this is the same patch as #7984

@ruchirK ruchirK enabled auto-merge August 25, 2021 17:10
@ruchirK ruchirK merged commit 45a3d2b into MaterializeInc:main Aug 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants