Persistent "no open shard found on ingester" after some indexer restart

[ncoiffier-celonis]

Describe the bug

After some indexer restart (for example auto-scaling/k8s repacking events), I'm sometimes getting a persistent stream of no open shard found on ingester errors. A rolling restart of all the indexers sometimes appears to fix it.

Steps to reproduce (if applicable)

Observed on a production environment (the restart happened at 7:03 CET/5:03 UTC):

Attached selected logs at the time of the restart (filtering out all the ingestion traffic, focusing on cluster events):

extract-2026-06-19T08_26_57.708Z.csv

Expected behavior

The no open shard found on ingester errors don't persist forever.

Configuration:

1. quickwit version: 0.9.0 (x86_64-unknown-linux-gnu 2026-04-19T08:54:33Z e1732a7) (the latest edge from month ago)
2. a few indexers, a single index taking 99% of the traffic, min_shards: 2.
3. constant traffic (i.e. the shard almost never idle and closed by CloseIdleShardsTask)
4. almost no calls to get_or_create_open_shards (1 call/ hour)

Tentative analysis:

I'm not 100% sure, but the logs would seem to point at some stalled routing tables in the other indexers

[ncoiffier-celonis]

1. Potentially similar to what was described in this comment: Routing table retains stale entries for short-lived shards #6324 (comment)?
2. I wonder if b33dd0f or 74b39ac or 4971fd6 would help? (happy to re-test if someone pushes a new edge version)
3. there is also this PR, not sure if we want to pursue this approach fix: drop stale routing entries on NoShardsAvailable failures osyniakov/quickwit#2 ?
4. maybe a better approach would be some kind of TTL on the routing table entries? Thoughts?