fix(telemetry): cold container-stats cache reports no_data, not running_count 0 (#1617)#1625
Merged
Merged
Conversation
…ng_count 0 (#1617) GET /api/telemetry/containers serves a per-uvicorn-worker, stale-while- revalidate in-process cache. The cold-cache branch returned `_empty_payload(0)` — a literal `running_count: 0` with `stale: true` — which is byte-identical to a healthy "zero containers running" answer. Because the cache is per worker, consecutive requests could flip between 0 and the true count depending on which worker served them, and a rarely-queried worker kept a cold cache indefinitely. In a real incident this was read as "the backend lost its Docker connection" and drove a wrong root-cause report. Fix (contract, no behavior change to the measured paths): - Cold cache now returns `running_count: null` + `no_data: true` so consumers can tell "not yet measured" from a real zero. The genuinely-empty fleet (refresh ran, found nothing running) still returns `running_count: 0` / `no_data: false`. - Add `no_data: false` to the fresh + stale branches for a consistent contract. - Document that `stale` refers to CACHE FRESHNESS only, not Docker connectivity (unreachable Docker is the existing HTTP 503). Redis-shared cache (the issue's optional "consider") is intentionally deferred — the misleading-diagnostics defect is fully closed by making cold state self-describing; a shared cache is a larger, separate change. Tests: update the #1096 cold-start unit test to the new shape, add a unit test proving cold(no_data)/None is distinct from a refreshed real zero, and make the live-API telemetry tests poll past the cold window. Related to #1617 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Resolve by running |
vybe
approved these changes
Jul 15, 2026
vybe
left a comment
Contributor
There was a problem hiding this comment.
Validated via /validate-pr.
Contract change is safe: I grepped the whole repo — nothing consumes /api/telemetry/containers outside the tests. The frontend running_count hits (stores/executions.js, network.js, OverviewPanel.vue) all read /api/executions/stats, a different endpoint. So making running_count nullable breaks no consumer.
no_data: true/running_count: nullcorrectly distinguishes 'not yet measured' from a real zero;test_cold_no_data_is_distinct_from_real_zero_runningproves both sides of that.- Named regression test present for the reported issue.
Security scan clean.
Two notes, neither blocking:
- Body edited — it read 'suggested fixes #1', which the issue-status-on-merge workflow regex (
/\b(?:fix(?:e[sd])?|close[sd]?|resolve[sd]?)\s+#(\d+)/gi) matched as a closing ref to #1 (a long-merged PR), and would have stamped status-in-dev on it while leaving #1617 stranded in status-in-progress. Now reads 'suggested fix 1' +Fixes #1617. - Follow-up:
docs/memory/feature-flows/host-telemetry.md:146documents this endpoint's response schema and still showsrunning_countas always-int with nono_datafield. Not merge-blocking (bug-fix tier), but the flow doc is now stale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /api/telemetry/containersserves a per-uvicorn-worker, stale-while-revalidate in-process cache. The cold-cache branch returned_empty_payload(0)— a literalrunning_count: 0withstale: true— which is byte-identical to a healthy "zero containers running" answer. Because the cache is per worker, consecutive requests could flip between0and the true count depending on which worker served them, and a rarely-queried worker kept a cold cache indefinitely (observedcache_age_seconds: 5726). In a real incident this read as "the backend lost its Docker connection" and drove a wrong root-cause report (#1617).Fix
Make the cold state self-describing so it can't be mistaken for a real zero:
running_count: null+no_data: true.running_count: 0+no_data: false(unchanged).no_data: falsefor a consistent contract.stale= cache freshness only, not Docker connectivity (unreachable Docker is the existing HTTP 503).Directly implements suggested fix 1 (
running_count: null/ explicitno_data) and #3 (documentstalesemantics) from the issue.Deferred (issue's optional "consider"): moving the cache to Redis for cross-worker consistency. The misleading-diagnostics defect is fully closed by making cold state self-describing; a shared-cache rework is a larger, separate change and out of scope for a P3 diagnostic-quality fix.
Consumer safety
No in-repo consumer reads
/api/telemetry/containers— the frontend uses only/api/telemetry/host; therunning_countreferences inOverviewPanel/network.js/executions.jscome from/api/executions/stats(a different endpoint/model). The endpoint's only known consumer is external diagnostic tooling (per the issue), so widening the contract withno_data+ a nullablerunning_countis backward-additive.Changes
src/backend/routers/telemetry.py: cold branch →running_count: null+no_data: true;no_data: falseon fresh/stale; docstring clarifiesstalesemantics.tests/unit/test_1096_telemetry_cache.py: update cold-start expectation to the new shape; newtest_cold_no_data_is_distinct_from_real_zero_runningproving cold≠real-zero.tests/test_telemetry.py: live-API tests poll past the cold window (skip/tolerateno_data).Test Plan
pytest tests/unit/test_1096_telemetry_cache.py -q→ 12 passedtests/test_telemetry.py::TestContainerTelemetry(test-runner suite, needs a running stack)Fixes #1617
🤖 Generated with Claude Code