fix: wait for exact-base quality artifacts (Fixes #516)

[Karthik Nadig (karthiknadig)]

Summary

Prevent false-negative quality checks when a PR starts before its exact-base performance or coverage baseline artifact has been published. A shared helper now waits for the exact SHA's successful push workflow and required artifact without falling back to moving main.

Changes

• poll exact-base baseline workflow runs and artifact publication with bounded timeout
• fail immediately when the exact-base workflow concludes unsuccessfully
• fail clearly on timeout, missing artifact, malformed API response, or missing token
• wait before both performance and coverage artifact downloads
• keep comparison steps fail-closed when waiting or downloading fails
• cover pending, success, failure, timeout, missing/expired artifact, run-selection, and input-validation paths

Validation

• python -m unittest discover -s scripts/tests -p 'test_*.py' -v (41 passed)
• both updated workflow YAML files parse successfully
• real exact-base performance and coverage artifact probes succeed

Fixes #516

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	1ms	+0ms	+0.0%	>5ms and >100%	➖
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Full refresh P50	65ms	60ms	+5ms	+8.3%	>25ms and >30%	🔺
Full refresh P95	68ms	62ms	+6ms	+9.7%	>50ms and >50%	🔺
Time to first environment P50	14ms	14ms	+0ms	+0.0%	>20ms and >100%	➖
Time to first environment P95	15ms	16ms	-1ms	-6.2%	>25ms and >100%	✅
Cold refresh P50	151ms	156ms	-5ms	-3.2%	>100ms and >50%	✅

Workload	PR	Baseline
Environments	5	5
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	101ms	65ms	+36ms	+55.4%	>100ms and >50%	🔺
Server startup P95	1028ms	706ms	+322ms	+45.6%	>750ms and >100%	🔺
Full refresh P50	172ms	112ms	+60ms	+53.6%	>100ms and >50%	🔺
Full refresh P95	234ms	137ms	+97ms	+70.8%	>300ms and >100%	🔺
Time to first environment P50	121ms	92ms	+29ms	+31.5%	>150ms and >50%	🔺
Time to first environment P95	169ms	109ms	+60ms	+55.0%	>250ms and >100%	🔺
Cold refresh P50	453ms	337ms	+116ms	+34.4%	>250ms and >50%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	80.815%	80.815%	+0.000pp
Functions	84.806%	84.806%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	8ms	8ms	+0ms	+0.0%	>10ms and >50%	➖
Server startup P95	11ms	11ms	+0ms	+0.0%	>50ms and >100%	➖
Full refresh P50	155ms	137ms	+18ms	+13.1%	>50ms and >30%	🔺
Full refresh P95	162ms	144ms	+18ms	+12.5%	>250ms and >100%	🔺
Time to first environment P50	21ms	17ms	+4ms	+23.5%	>25ms and >50%	🔺
Time to first environment P95	30ms	30ms	+0ms	+0.0%	>100ms and >100%	➖
Cold refresh P50	157ms	138ms	+19ms	+13.8%	>150ms and >50%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	2	2

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	78.207%	78.207%	+0.000pp
Functions	81.533%	81.533%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[Copilot]

Pull request overview

This PR adds a shared, exact-SHA baseline “wait” helper to prevent false-negative quality checks when performance/coverage comparison jobs start before the base commit’s baseline artifacts are published. It integrates the helper into the PR performance and coverage workflows so they wait (with a bounded timeout) for a successful exact-base run and the required artifact, failing clearly if the workflow fails or artifacts never appear.

Changes:

• Added scripts/wait_for_baseline.py to poll for the exact-base push workflow run and required artifact with timeout, clear errors, and fail-fast on unsuccessful conclusions.
• Added unit tests covering polling behavior, run selection, input validation, timeout paths, and artifact filtering.
• Updated performance and coverage workflows to wait before downloading exact-base artifacts.

Show a summary per file

File	Description
scripts/wait_for_baseline.py	New helper that waits for an exact-commit baseline workflow + artifact via GitHub API with bounded polling and clear failure modes.
scripts/tests/test_wait_for_baseline.py	New unittest coverage for polling, failures, timeouts, artifact selection, and input validation.
.github/workflows/perf-tests.yml	Adds a “wait for exact PR base performance” step prior to baseline artifact download.
.github/workflows/coverage.yml	Adds a “wait for exact PR base coverage” step prior to baseline artifact download.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite