Latency only on one distributor

[friedrichg]

Describe the bug
There is increased high latency in one distributor:

To Reproduce

Not easy. I noticed the problem upgrading a cluster from v1.16.1 to v1.17.1.
After the distributors finish deploying the problem appers.

I was able to track down that the problem started in
cf574c9
and is not present in
f9cb3ea

Expected behavior
There should not be high latency in one distributor

Environment:

• Infrastructure: Kubernetes
• Deployment tool: k8s

Additional Context
I am not able to reproduce this problem in smaller cluster with less than 100 ingesters.

[friedrichg]

I can confirm the problem is gone after 63677ef