compute: mz_dataflow_initial_output_duration_seconds

[teskje]

This PR introduces a new replica metric, mz_dataflow_initial_output_duration_seconds, that tracks the time from the installation of a compute collection until it first produced any outputs. This is not necessarily the time until a dataflow has been fully hydrated (depending on your definition of 'hydration'), but might be a good stand-in.

From the design doc (#19717):

• mz_dataflow_initial_output_duration_seconds
  • Type: gauge
  • Labels: worker_id, collection_id
  • Description: The time from dataflow installation up to when the first output was produced.
  • Export Type: direct
  • Notes: To reduce the cardinality of this metric, we limit it to non-transient dataflows.

Demo

You can try the metric out by running these commands in psql:

materialize=> create table t (a int);
CREATE TABLE
materialize=> insert into t select generate_series(1, 2000);
INSERT 0 2000
materialize=> create view v as select t1.a as x, t2.a as y from t t1, t t2;
CREATE VIEW
materialize=> \timing
Timing is on.
materialize=> create default index on v; select * from v limit 1;
CREATE INDEX
Time: 173.885 ms
  x  |  y
-----+-----
 256 | 256
(1 row)

Time: 15370.698 ms (00:15.371)

Then check the metrics endpoint and you should see an entry with a time close to the duration of that final SELECT:

mz_dataflow_initial_output_duration_seconds{collection_id="u2",worker_id="0"} 15.362563792

When you drop the index, the metrics entry will go away too.

Motivation

• This PR adds a known-desirable feature.

Part of MaterializeInc/database-issues#5547.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered.
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• This PR includes the following user-facing behavior changes:
  • N/A

[teskje]

Made private to make it more likely that the different paths that update the reported frontier also call observe_snapshot_produced. Unfortunately, most of these paths are in the same module and there is nothing but discipline keeping us from modifying reported_frontier directly.

[antiguru]

Thanks, I think to looks fine. I wonder if we can improve the testing of this PR. If I recall correctly, we don't have a way to test the metrics exported by a replica, so it's not easy to do this at the moment. Maybe we should invest a bit more in this? (Or, file an issue so we don't forget!)

[teskje]

Added an mzcompose-based cluster test. And it found a bug, so huge thanks for insisting on a test @antiguru!

[vmarcos]

This will be extremely helpful, thanks!

[teskje]

TFTRs!