Ingester trying to open a TSDB block index reader while shutting down

[pracucci]

Describe the bug
Today I experienced the following log while an ingester was shutting down after getting the /shutdown endpoint called:

Issue: `level=warn ts=2020-10-14T08:35:20.077984655Z caller=grpc_logging.go:55 method=/cortex.Ingester/QueryStream duration=11.567608ms err="open querier for block 01EMJ919X41XRBW40XW40DDVZA: open index reader: block is closing" msg="gRPC\n"`

Expected behavior
We should add a circuit breaker to make sure no read occurs on TSDBs once Ingester.closeAllTSDB() has been called.

Storage Engine

• Blocks
• Chunks

[pstibrany]

Is this a problem? Looks like correct behavior to me.

[pracucci]

Is this a problem? Looks like correct behavior to me.

This log occurs only if the query is received while TSDB is closing. The query may be receive even after a TSDB is closed but before ingester has terminated, which leads to TSDB getting reopened.

I believe that we should have a circuit breaker to make sure we don't try to re-open a TSDB (and ideally even not querying it) once the ingester reaches closeAllTSDB().

What do you think?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

I think it's still valid

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[bboreham]

I marked it "good first issue" as it shouldn't be too hard to add a check whether we are closing down before trying to execute a query.

However I think we will still log some error message; we have to return failure to the caller, and this log line comes from gRPC.
Maybe have a class of errors that are not worth logging as gRPC errors?

[ilangofman]

Hi, I am trying to gain a better understanding of this issue. Once the Ingester.closeAllTSDB() function has been called and all the TSDB's have been deleted from the TSDBState.dbs map, wouldn't the query return an empty response since db := i.getTSDB(userID) should return nil. From the ingester query functions, the TSDB should not be created if it does not already exist in the map.

I guess whats missing from my understanding right now is how the TSDB can be re-opened once it has been deleted from the map in TSDBState.

Thank you in advance for the help.

[bboreham]

Once the Ingester.closeAllTSDB() function has been called and all the TSDB's have been deleted from the TSDBState.dbs map

The delete only happens after all of them have closed, due to waiting on the mutex.
Hence "The query may be receive even after a TSDB is closed but before ingester has terminated" (added emphasis).