Skip to content
This repository was archived by the owner on Jun 19, 2026. It is now read-only.

receive: Add TSDB commit duration metric to receive writer#307

Merged
yishi3-db merged 2 commits into
db_mainfrom
yi-shi_data/yishi3-db/add-commit-duration-metric
Mar 6, 2026
Merged

receive: Add TSDB commit duration metric to receive writer#307
yishi3-db merged 2 commits into
db_mainfrom
yi-shi_data/yishi3-db/add-commit-duration-metric

Conversation

@yishi3-db

@yishi3-db yishi3-db commented Mar 6, 2026

Copy link
Copy Markdown

Changes

Add a new Prometheus histogram metric thanos_receive_writer_commit_duration_seconds to track
the latency of TSDB Commit() calls in the receive writer.

Previously there was no way to observe how quickly metric samples are persisted to disk.
The existing thanos_receive_write_e2e_latency_seconds covers full end-to-end ingestion latency
but does not isolate the disk commit step. This metric makes that visible so we can track
changes in storage performance over time.

Verification

  1. Tested with unit tests (go test -run TestWriter ./pkg/receive/...).
  2. Passed CI
  3. Verified in integtest env, saw metrics got emitted

@yishi3-db yishi3-db marked this pull request as ready for review March 6, 2026 21:28

@willh-db willh-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

@yishi3-db yishi3-db merged commit f0e0ee3 into db_main Mar 6, 2026
14 checks passed
@yishi3-db yishi3-db deleted the yi-shi_data/yishi3-db/add-commit-duration-metric branch March 6, 2026 23:28
@yuchen-db

Copy link
Copy Markdown
Collaborator

Seems to be a high cardinality metric?

@willh-db

willh-db commented Mar 7, 2026

Copy link
Copy Markdown
Collaborator

Seems to be a high cardinality metric?

@yuchen-db cardinality should be (16 histogram buckets) * (# of db pods) so thanos_receive_writer_commit_duration_seconds_bucket has an upper bound of ~15,000 time series in the biggest clusters unless I am missing something

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants