Metrics for tsdb startup time

[damnever]

Is your feature request related to a problem? Please describe.
Add some metrics to show the tsdb startup time.

Describe the solution you'd like
We can reuse the tsdb level metrics, such as prometheus_tsdb_data_replay_duration_seconds and prometheus_tsdb_snapshot_replay_error_total. Alternatively, we can also introduce new metrics at the cortex level specific to each tenant.

[yeya24]

I like this. I guess it is fine to make it per tenant since we have other per tenant metrics in ingester anyway

[damnever]

However, there is already a metric called cortex_ingester_tsdb_wal_replay_duration_seconds

cortex/pkg/ingester/ingester.go

Lines 590 to 594 in 634df35

	walReplayTime: promauto.With(registerer).NewHistogram(prometheus.HistogramOpts{
	Name: "cortex_ingester_tsdb_wal_replay_duration_seconds",
	Help: "The total time it takes to open and replay a TSDB WAL.",
	Buckets: prometheus.DefBuckets,
	}),

[yeya24]

cortex_ingester_tsdb_wal_replay_duration_seconds this one seems a little bit weird as it is the total time of opening a tsdb, including wal replay and other time.
Does it make sense to rename the metric?

[damnever]

Perhaps we should deprecate cortex_ingester_tsdb_wal_replay_duration_seconds and replace it with cortex_ingester_tsdb_data_replay_duration_seconds . Since I personally do not find the percentile metric useful for identifying slow users when considering related context information such as the number of series the user has.

[yeya24]

I think I am ok to align with TSDB metrics. Thoughts? @friedrichg @alanprot @alvinlin123 ?