per-tenant metrics on chunks-per-query, chunk-size-bytes-per-query, and fetched-series-per-query

[alvinlin123]

Is your feature request related to a problem? Please describe.
To better understand what values we can set for each user for querier.max-fetched-chunks-per-query, querier.max-fetched-series-per-query, and querier.max-fetched-chunk-bytes-per-query, it would be invaluable to see the user's current number on fetched-chunks-per-query, fetched-chunk-bytes-per-query, and fetched-series-per-query.

Describe the solution you'd like
per-tenant and global histogram metrics for fetched-chunks-per-query, fetched-chunk-bytes-per-query, and fetched-series-per-query on querier side.

Describe alternatives you've considered
Trial and error.

Additional context
N/A

[tomwilkie]

(As discussed on a call)

• need to think very carefully about any per-user metrics, they are very expensive.
• can't have per-user histograms, the cardinality is just too high
• could sample a gauge (and use a very long average_over_time) but its noisey and not very clear what signal you're getting
• could have a sum and a count metric, and have accurate averages - but not clear what the avg is telling you.
• could use a summary exporting a single quantile (0.99) as a pseudo-moving-max
• personally, I would stick this in the logs.