From 84a27015a0600f041337fb2b3348a6303a31a3f4 Mon Sep 17 00:00:00 2001 From: Yordis Prieto Date: Thu, 2 Apr 2026 23:42:29 -0400 Subject: [PATCH] feat(trogonstack-datadog): add dashboard naming, telemetry exploration, and audit workflow Signed-off-by: Yordis Prieto --- .../.claude-plugin/plugin.json | 2 +- plugins/trogonstack-datadog/README.md | 2 +- .../skills/datadog-design-dashboard/SKILL.md | 267 +++++++++++++++--- .../references/layouts.md | 17 +- .../references/thresholds.md | 28 ++ .../skills/datadog-review-dashboard/SKILL.md | 232 --------------- 6 files changed, 268 insertions(+), 280 deletions(-) delete mode 100644 plugins/trogonstack-datadog/skills/datadog-review-dashboard/SKILL.md diff --git a/plugins/trogonstack-datadog/.claude-plugin/plugin.json b/plugins/trogonstack-datadog/.claude-plugin/plugin.json index adc2d9c..bb4c642 100644 --- a/plugins/trogonstack-datadog/.claude-plugin/plugin.json +++ b/plugins/trogonstack-datadog/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "trogonstack-datadog", - "description": "Datadog observability skills for designing dashboards with proper widget selection, layout patterns, and template variables using the pup CLI", + "description": "Datadog observability skills for designing, redesigning, and auditing dashboards with proper widget selection, layout patterns, template variables, and operational readiness validation using the pup CLI", "version": "0.1.0", "author": { "name": "TrogonStack", diff --git a/plugins/trogonstack-datadog/README.md b/plugins/trogonstack-datadog/README.md index 6410dd7..60687e5 100644 --- a/plugins/trogonstack-datadog/README.md +++ b/plugins/trogonstack-datadog/README.md @@ -1,6 +1,6 @@ # trogonstack-datadog -Datadog observability skills for designing dashboards with proper widget selection, layout patterns, and template variables using the pup CLI. +Datadog observability skills for designing, redesigning, and auditing dashboards with proper widget selection, layout patterns, template variables, and operational readiness validation using the pup CLI. ```bash claude plugin install trogonstack-datadog@trogonstack diff --git a/plugins/trogonstack-datadog/skills/datadog-design-dashboard/SKILL.md b/plugins/trogonstack-datadog/skills/datadog-design-dashboard/SKILL.md index 8ad379a..5f0fe7b 100644 --- a/plugins/trogonstack-datadog/skills/datadog-design-dashboard/SKILL.md +++ b/plugins/trogonstack-datadog/skills/datadog-design-dashboard/SKILL.md @@ -1,12 +1,15 @@ --- name: datadog-design-dashboard description: >- - Design Datadog dashboards with proper widget selection, layout organization, - template variables, and group structure. Uses pup CLI for inspecting existing - dashboards and validating designs. Use when designing new dashboards, auditing - existing ones, or planning observability layouts for services. Do not use for: - (1) Datadog agent installation or configuration, (2) monitor/alert rule design, - (3) APM instrumentation or tracing setup, (4) log pipeline configuration. + Design new Datadog dashboards, redesign existing ones, or audit dashboards + for operational readiness. Covers widget selection, layout organization, + template variables, group structure, alert threshold validation, and + zero-knowledge readability. Uses pup CLI for inspecting dashboards and + validating designs. Use when designing new dashboards, redesigning existing + ones, auditing before on-call handoff, or reviewing after dashboard changes. + Do not use for: (1) Datadog agent installation or configuration, + (2) monitor/alert rule design, (3) APM instrumentation or tracing setup, + (4) log pipeline configuration. allowed-tools: AskUserQuestion, Write, Read, Shell --- @@ -18,8 +21,43 @@ Design a dashboard layout that tells a clear story — from high-level health si **Philosophy**: The frameworks, layouts, and widget guides in this skill are starting points — not rigid rules. Every product and business is different. Understand the domain first, then adapt the frameworks to fit. The best dashboards reflect how the business actually works, not how a generic template says they should. +--- + +## Interview + +First, determine the mode: +- **Design mode** — user wants to create or redesign a dashboard → ask design questions, then run the full workflow +- **Audit mode** — user wants to review an existing dashboard with no intent to redesign → skip to [Audit](#audit) + +### Design Questions + +**Skip if ALL of these are already specified**: dashboard purpose, target audience, data sources, template variable needs, dashboard strategy. + +1. **Purpose** — "What is this dashboard for? Service overview, infrastructure, executive KPIs, debugging, or SLO tracking?" +2. **Audience** — "Who will use this? On-call engineers, platform team, leadership, or mixed?" +3. **Data Sources** — "Which Datadog products are involved? Metrics only, APM + Metrics, Logs + Metrics, or full stack?" +4. **Scope** — "Is this for a single service, a group of services, or infrastructure-wide?" +5. **Dashboard Strategy** — "One dashboard per service, or a consolidated view?" — share the trade-offs from [references/layouts.md](references/layouts.md) to help them decide. If unsure, ask: "During an outage, does your team investigate one service at a time, or do they need to see all services simultaneously?" +6. **Existing Dashboard** — "Is there an existing dashboard to audit or redesign?" If yes, fetch with `pup dashboards get ` before designing. + +### Audit Questions + +**Skip if ALL of these are already specified**: dashboard ID or URL, service name or team context. + +**Always interview if**: No dashboard ID is provided or multiple dashboards may be relevant. + +1. **Dashboard** — "Which dashboard should I review? Provide a dashboard ID, URL, or service name to search for." +2. **Business Context** — "Can you tell me what this service does for customers? Are there codebases or docs I can read to understand the product?" + - Impact: Understanding the domain lets the review focus on whether the right metrics are being tracked, not just whether generic rules are followed +3. **Focus** — "Is there anything specific you want me to focus on? (A) Full review, (B) Alert thresholds only, (C) Business section, (D) Layout and readability" + - Default to full review if unspecified + +--- + ## Domain Discovery +*Applies to design mode. Skip if auditing only.* + Before designing, understand what you are building observability for. The metrics that matter depend entirely on the product and business context. **Ask the user**: @@ -33,34 +71,36 @@ Before designing, understand what you are building observability for. The metric **Skip domain discovery if**: You already have deep context about the service from prior conversations or the user has provided detailed specifications. -## Interview - -**Skip if ALL of these are already specified**: dashboard purpose, target audience, data sources, template variable needs, dashboard strategy. - -**Always interview if**: Auditing or redesigning an existing dashboard (needs current state review first). - -1. **Purpose** — "What is this dashboard for? Service overview, infrastructure, executive KPIs, debugging, or SLO tracking?" -2. **Audience** — "Who will use this? On-call engineers, platform team, leadership, or mixed?" -3. **Data Sources** — "Which Datadog products are involved? Metrics only, APM + Metrics, Logs + Metrics, or full stack?" -4. **Scope** — "Is this for a single service, a group of services, or infrastructure-wide?" -5. **Dashboard Strategy** — "One dashboard per service, or a consolidated view?" — share the trade-offs from [references/layouts.md](references/layouts.md) to help them decide. If unsure, ask: "During an outage, does your team investigate one service at a time, or do they need to see all services simultaneously?" -6. **Existing Dashboard** — "Is there an existing dashboard to audit or redesign?" If yes, fetch with `pup dashboards get ` before designing. - --- -## Workflow +## Design + +*Skip to [Audit](#audit) if the user only wants to review an existing dashboard.* ### 1. Gather existing context ```bash pup dashboards list pup dashboards get -pup dashboards url ``` If auditing an existing dashboard, fetch its definition first and analyze its current structure before redesigning. -### 2. Choose a framework +### 2. Explore available telemetry + +Before designing widgets, check what metrics and tag values actually exist for the service. This prevents designing around metrics that don't exist or using the wrong tag values in queries. + +```bash +# See what metrics are available for the service +pup metrics list --filter=".*" + +# Verify the service tag is active and see what metrics are flowing +pup metrics list --filter="trace.*" --tag-filter="service:" +``` + +Use the actual metric names and tag values you find here when writing widget queries — do not guess or invent them. If a metric you expect does not appear, flag it to the user before building widgets around it. + +### 3. Choose a framework Match the dashboard purpose to a framework. Read [references/frameworks.md](references/frameworks.md) for detailed metric mappings and group structures. @@ -72,22 +112,15 @@ Match the dashboard purpose to a framework. Read [references/frameworks.md](refe | SLO tracking | SLI/SLO | | Debugging | Drill-down | -### 3. Design the layout +### 4. Design the layout -Using your domain understanding and the chosen framework, design the group structure and select widgets. Read these references as needed: +Using your domain understanding and the chosen framework, design the group structure and select widgets. Read these references before designing: - **[layouts.md](references/layouts.md)** — Template variable conventions, group structure patterns, dashboard strategy trade-offs, grid sizing, anti-patterns - **[widgets.md](references/widgets.md)** — Widget selection guide, display options, sizing, naming conventions - **[thresholds.md](references/thresholds.md)** — Alert threshold markers, threshold proximity, Y-axis configuration -**Key principles** (not rigid rules — use judgment): -- **Prefix every widget title** with its layer and priority: `I0:` (most critical infra), `P0:` (most critical platform), `D0:` (most critical domain), `B0:` (most critical business). See [widgets.md](references/widgets.md) for the full prefix system. -- Start with a **Business** group (5-8 `B`-prefixed metrics) so someone with zero service knowledge can tell if customers are affected within 5 seconds. Tailor the metrics to the domain. -- Timeseries widgets should have **alert threshold markers** (red lines) with thresholds close to normal traffic. If a metric doesn't warrant an alert, question whether it belongs — but context-providing metrics can earn their place. -- Set **Y-axis max** explicitly near the threshold — don't let auto-scaling compress the normal range. -- Order groups macro-to-micro: business → overview → domain-specific → infrastructure. - -### 4. Write the output +### 5. Write the design output Present the design using this template: @@ -110,38 +143,188 @@ Present the design using this template: | ... | ... | ... | ... | ... | [Repeat for each group] - -## Quality Validation -[Run quality principles below] ``` -### 5. Validate +--- + +## Audit + +Applies to both modes. Run after design, or directly if auditing an existing dashboard. + +The core principles are: graphs should earn their place with alert thresholds, thresholds should sit close to normal traffic, a business section should exist at the top, and the dashboard should be readable by someone with zero service knowledge. + +These are guiding principles — not a rigid checklist. Apply judgment based on the product and business context. A context-providing metric (like deployment events) may earn its place without a threshold. A service with unusual traffic patterns may need different proximity rules. + +### 1. Fetch the dashboard ```bash +# If given a service name, list all dashboards and identify the relevant one by title pup dashboards list + +# If given a URL, extract the dashboard ID from the path (e.g., /dashboard/abc-def-ghi/...) + +# Get the full dashboard definition (includes the dashboard URL in the response) pup dashboards get + +# Verify real metric names exist pup metrics list --filter="trace.http.request.*" ``` +Parse the response to build an inventory of all widgets, groups, and their configurations. + +### 2. Build widget inventory + +Read [references/widgets.md](references/widgets.md) for the full widget prefix system before cataloging. + +Catalog every widget in the dashboard: + +| Widget Title | Prefix | Type | Group | Has Alert Threshold | Threshold Value | Notes | +|-------------|--------|------|-------|--------------------:|----------------|-------| +| ... | I0/P1/D0/B0/— | ... | ... | ... | ... | ... | + +Focus on timeseries and query value widgets — these are the primary candidates for alert threshold markers. + +### 3. Audit alert thresholds + +Read [references/thresholds.md](references/thresholds.md) for threshold marker principles, configuration details, and findings format. + +For each timeseries widget, check: +- Does it have a marker/threshold line configured? +- Is the marker colored red for visibility? +- Does the threshold correspond to an actual monitor/alert? + +### 4. Audit threshold proximity + +Read [references/thresholds.md](references/thresholds.md) for proximity guidance, Y-axis configuration rules, and findings format. + +For each widget with a threshold, check: +- What is the typical (normal) value range? +- Where is the threshold set? +- Is there excessive whitespace between the normal line and the alert line? +- Is the Y-axis auto-scaled or explicitly set? + +### 5. Audit business section + +**Principle**: A dedicated Business (`B`) group should exist at the top of the dashboard with 5-8 key metrics for immediate outage identification. Business metrics are customer-visible outcomes — not infrastructure or domain internals. The specific metrics should reflect the product's business transactions, not generic traffic and error rates. + +Check: +- Does a Business group exist (named "Business", "B", or equivalent)? +- Is it the first group on the dashboard? +- Do its widgets use the `B0-N:` prefix? +- Does it contain 5-8 metrics covering: customer-visible success rates, key transaction flows, and SLA-impacting latency? +- Can someone determine "are customers affected?" within 5 seconds of opening the dashboard? + +**Findings format**: + +```markdown +#### Business Section Audit + +**Status**: MISSING / INCOMPLETE / OK + +**Current state**: [Description of what exists] + +**Recommended metrics** (if missing or incomplete): +1. B0: Key transaction success rate (are critical flows completing?) +2. B0: Customer-facing error rate (are requests failing for customers?) +3. B1: API p99 latency (are responses slow for customers?) +4. B1: Total request rate (are we receiving traffic?) +5. B2: Queue depth or processing lag (is async work backing up?) +6. B2: Key business event throughput (e.g. orders created, payments processed) +``` + +### 6. Apply zero-knowledge viewer test + +**Principle**: Someone with zero knowledge of the service should be able to spot problems by looking for red indicators. + +Evaluate: +- Can you identify a problem in under 10 seconds without reading widget titles? +- Are thresholds visible as red lines on every graph? +- Is conditional formatting applied to query value widgets (green/yellow/red)? +- Are group names self-explanatory? +- Is there a note widget with runbook links or team ownership? + +**Findings format**: + +```markdown +#### Zero-Knowledge Readability Audit + +| Check | Status | Finding | +|-------|--------|---------| +| Problems visible in <10s | FAIL | No red lines on 8 of 12 graphs | +| Conditional formatting on QV widgets | PARTIAL | 2 of 4 QV widgets have thresholds | +| Group names self-explanatory | OK | All groups use clear names | +| Runbook/ownership note | MISSING | No note widget with team info | +``` + +### 7. Generate audit report + +Compile all findings into a structured report: + +```markdown +# Dashboard Audit: [Dashboard Title] + +**Dashboard ID**: [id] +**URL**: [url] +**Review date**: [date] + +## Summary + +[2-3 sentence summary: overall health of the dashboard, critical issues count] + +## Critical Issues + +[List issues that must be fixed before the dashboard is production-ready] + +## Alert Threshold Audit +[From step 3] + +## Threshold Proximity Audit +[From step 4] + +## Business Section Audit +[From step 5] + +## Zero-Knowledge Readability Audit +[From step 6] + +## Recommended Actions + +### Must Fix +1. [Action item with specific widget and group reference] + +### Should Fix +1. [Action item] + +### Nice to Have +1. [Action item] +``` + --- ## Quality Principles +- [ ] Widget queries use real metric names verified via `pup metrics list` — no invented metric names - [ ] Dashboard reflects the actual product and business — metrics tailored to the domain -- [ ] Dashboard title is concise (no environment, region, or version) -- [ ] Template variables defined for env, service, and relevant scopes (default `*`) +- [ ] Dashboard title follows `[service] Purpose` pattern — no "Dashboard" suffix, no environment in the title +- [ ] `title` field updated in the JSON (not just the filename) — redeploy after any title change +- [ ] Template variables match the dashboard type — see [references/layouts.md](references/layouts.md) +- [ ] Widget queries use template variable scopes verified via `pup metrics list` — no hardcoded env, service, or host values; use the variable set appropriate for the dashboard type (see [references/layouts.md](references/layouts.md)) - [ ] **Business group** with 5-8 `B`-prefixed metrics tailored to the service's customer-visible outcomes - [ ] Groups ordered macro-to-micro (business → overview → details) -- [ ] **Timeseries widgets have alert threshold markers** (red lines) where the metric is alertable +- [ ] **Every widget title uses the layer-priority prefix** (`I0:`, `P1:`, `D0:`, `B0:`, etc.) — see [references/widgets.md](references/widgets.md) +- [ ] Widget titles use sentence case, don't repeat group name +- [ ] **Timeseries widgets have alert threshold markers** (red lines) where the metric is alertable — see [references/thresholds.md](references/thresholds.md) - [ ] **Thresholds close to normal traffic** — no excessive whitespace - [ ] **Zero-knowledge readability** — someone with no service knowledge can spot problems via red indicators -- [ ] **Widget titles prefixed** with layer and priority (`I0:`, `P1:`, `D0:`, `B0:`, etc.) -- [ ] Widget titles use sentence case, don't repeat group name +- [ ] Query Value widgets have conditional formatting (green/yellow/red) - [ ] Every metric earns its place — if it spikes, someone can act on it +- [ ] All audit findings include specific widget names and group references +- [ ] Recommended actions categorized by priority (must/should/nice-to-have) +- [ ] Dashboard URL included in audit report ## References - **[Observability Frameworks](references/frameworks.md)** — RED, USE, Golden Signals, SLI/SLO with metric mappings - **[Layout & Structure](references/layouts.md)** — Template variables, group patterns, dashboard strategy, grid sizing, anti-patterns -- **[Widgets](references/widgets.md)** — Widget types, display options, sizing, naming conventions +- **[Widgets](references/widgets.md)** — Widget prefix system, types, display options, sizing, naming conventions - **[Alert Thresholds](references/thresholds.md)** — Threshold markers, proximity guide, Y-axis configuration diff --git a/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/layouts.md b/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/layouts.md index 93c8605..9abf3b7 100644 --- a/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/layouts.md +++ b/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/layouts.md @@ -8,17 +8,26 @@ Template variable conventions, group structure patterns, dashboard strategy trad Template variables make one dashboard serve many contexts. Define them before laying out widgets. -**Standard variables** (include when applicable): +**`env` is always required.** The remaining variables depend on the dashboard type: + +| Dashboard Type | Template Variables | +|---------------|-------------------| +| Service Overview (RED), Debugging | `env` · `service` | +| Infrastructure (USE) | `env` · `host` · `availability_zone` | +| Executive / Golden Signals | `env` · `team` · `region` | + +**Optional variables** (add when relevant to the dashboard type): | Variable | Tag | Use Case | |----------|-----|----------| -| `env` | `env` | Environment filtering (prod, staging, dev) | -| `service` | `service` | Service scoping | | `region` | `region` | Regional filtering | | `availability_zone` | `availability_zone` | AZ-level drill-down | | `host` | `host` | Host-level investigation | +| `endpoint` | `http.url` | Per-route investigation (Debugging dashboards) | -- Start with the broadest scope (`env`) and narrow down +- For **service dashboards** (RED, Debugging): scope every widget query with `{$service,$env}` +- For **infrastructure dashboards**: scope queries with `{$host,$env}` or `{$availability_zone,$env}` instead +- For **executive dashboards**: scope queries with `{$team,$env}` or `{$region,$env}` instead - Use `*` as the default value so dashboards load with full scope - Never put environment or region in the dashboard title — that is what template variables are for - Name variables after the tag they filter on diff --git a/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/thresholds.md b/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/thresholds.md index 567f0e8..7b41826 100644 --- a/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/thresholds.md +++ b/plugins/trogonstack-datadog/skills/datadog-design-dashboard/references/thresholds.md @@ -53,6 +53,34 @@ Do not rely on Y-axis auto-scaling. Set `yaxis.max` explicitly to slightly above --- +## Audit Findings Format + +### Alert Threshold Audit + +```markdown +#### Alert Threshold Audit + +| Widget | Group | Status | Finding | +|--------|-------|--------|---------| +| Requests/s | Rate | MISSING | No threshold marker — add alert line or remove widget | +| Error rate | Errors | OK | Red line at 5% | +| CPU usage | Infra | MISSING | No threshold — is this metric alertable? | +``` + +### Threshold Proximity Audit + +```markdown +#### Threshold Proximity Audit + +| Widget | Normal Range | Threshold | Gap | Y-Axis | Status | +|--------|-------------|-----------|-----|--------|--------| +| CPU usage | ~20% | 95% | 75% | auto | TOO FAR — lower to 40-50%, set Y-max to 55% | +| Error rate | ~0.1% | 5% | ~5% | auto | OK gap — but set Y-max to 6% | +| p99 latency | ~50ms | 500ms | 10x | auto | TOO FAR — lower to 100-150ms, set Y-max to 175ms | +``` + +--- + ## Metrics Near Zero Some metrics hover close to zero under normal conditions (e.g., error counts, retry rates, dead letter queue depth). For these: diff --git a/plugins/trogonstack-datadog/skills/datadog-review-dashboard/SKILL.md b/plugins/trogonstack-datadog/skills/datadog-review-dashboard/SKILL.md deleted file mode 100644 index 195461a..0000000 --- a/plugins/trogonstack-datadog/skills/datadog-review-dashboard/SKILL.md +++ /dev/null @@ -1,232 +0,0 @@ ---- -name: datadog-review-dashboard -description: >- - Review existing Datadog dashboards for operational readiness. Audits alert - threshold markers, threshold proximity to normal traffic, business section - completeness, and zero-knowledge readability. Uses pup CLI to fetch - dashboard definitions. Use when auditing dashboards before on-call handoff, - after dashboard changes, or during operational reviews. Do not use for: - (1) designing new dashboards from scratch, (2) monitor/alert rule design, - (3) APM instrumentation or tracing setup, (4) log pipeline configuration. -allowed-tools: AskUserQuestion, Read, Shell ---- - -# Review Datadog Dashboard - -Audit an existing Datadog dashboard against operational readiness principles. The core principles are: graphs should earn their place with alert thresholds, thresholds should sit close to normal traffic, a business section should exist at the top, and the dashboard should be readable by someone with zero service knowledge. - -These are guiding principles — not a rigid checklist. Apply judgment based on the product and business context. A context-providing metric (like deployment events) may earn its place without a threshold. A service with unusual traffic patterns may need different proximity rules. - -## Interview Phase - -**Skip interview if ALL of these are already specified:** -- Dashboard ID or URL -- Service name or team context - -**Always interview if**: No dashboard ID is provided or multiple dashboards may be relevant. - -### Questions - -1. **Dashboard** — "Which dashboard should I review? Provide a dashboard ID, URL, or service name to search for." - - Impact: Determines which dashboard definition to fetch - -2. **Business Context** — "Can you tell me what this service does for customers? Are there codebases or docs I can read to understand the product?" - - Impact: Understanding the domain lets the review focus on whether the right metrics are being tracked, not just whether generic rules are followed - -3. **Focus** — "Is there anything specific you want me to focus on? (A) Full review, (B) Alert thresholds only, (C) Business section, (D) Layout and readability" - - Impact: Determines review scope — default to full review if unspecified - ---- - -## Workflow - -### 1. Fetch Dashboard Definition - -```bash -# If given a service name, search for matching dashboards -pup dashboards list --filter="" - -# If given a URL, extract the dashboard ID from the path (e.g., /dashboard/abc-def-ghi/...) - -# Get the full dashboard definition -pup dashboards get - -# Get the dashboard URL for reference -pup dashboards url -``` - -Parse the response to build an inventory of all widgets, groups, and their configurations. - -### 2. Build Widget Inventory - -Catalog every widget in the dashboard: - -| Widget Title | Prefix | Type | Group | Has Alert Threshold | Threshold Value | Notes | -|-------------|--------|------|-------|--------------------:|----------------|-------| -| ... | I0/P1/D0/B0/— | ... | ... | ... | ... | ... | - -Check that every widget title uses the layer-priority prefix system: -- `I0-N:` for infrastructure (load balancers, databases, networks) -- `P0-N:` for platform (service-specific components from the codebase) -- `D0-N:` for domain (technical health of domain processes — tech stuff) -- `B0-N:` for business (business outcomes — business stuff) -- The number indicates priority within the layer (`0` = most critical) - -Focus on timeseries and query value widgets — these are the primary candidates for alert threshold markers. - -### 3. Audit Alert Thresholds - -**Principle**: Timeseries graphs should generally have an alert threshold (red line). If a metric doesn't warrant an alert, question whether it belongs — but use judgment. Some metrics provide valuable context (deployment markers, dependency traffic patterns) without needing a threshold. - -For each timeseries widget, check: -- Does it have a marker/threshold line configured? -- Is the marker colored red for visibility? -- Does the threshold correspond to an actual monitor/alert? - -**Findings format**: - -```markdown -#### Alert Threshold Audit - -| Widget | Group | Status | Finding | -|--------|-------|--------|---------| -| Requests/s | Rate | MISSING | No threshold marker — add alert line or remove widget | -| Error rate | Errors | OK | Red line at 5% | -| CPU usage | Infra | MISSING | No threshold — is this metric alertable? | -``` - -### 4. Audit Threshold Proximity - -**Principle**: Alert thresholds must be close to normal traffic. Large gaps between normal values and the alert line create blind spots where anomalies go unnoticed. - -For each widget with a threshold: -- What is the typical (normal) value range? -- Where is the threshold set? -- Is there excessive whitespace between the normal line and the alert line? -- Is the Y-axis auto-scaled or explicitly set? Auto-scaled Y-axes compress normal traffic into a flat band when the threshold is far above normal — the Y-axis max should be set to slightly above the alert threshold - -**Bad example**: Normal CPU is 20%, alert threshold at 95% — the graph is mostly empty space and a slow climb from 20% to 80% looks flat. - -**Good example**: Normal CPU is 20%, alert threshold at 45% — anomalies visually stand out immediately. - -**Findings format**: - -```markdown -#### Threshold Proximity Audit - -| Widget | Normal Range | Threshold | Gap | Y-Axis | Status | -|--------|-------------|-----------|-----|--------|--------| -| CPU usage | ~20% | 95% | 75% | auto | TOO FAR — lower to 40-50%, set Y-max to 55% | -| Error rate | ~0.1% | 5% | ~5% | auto | OK gap — but set Y-max to 6% | -| p99 latency | ~50ms | 500ms | 10x | auto | TOO FAR — lower to 100-150ms, set Y-max to 175ms | -``` - -### 5. Audit Business Section - -**Principle**: A dedicated Business (`B`) group should exist at the top of the dashboard with 5-8 key metrics for immediate outage identification. Business metrics are customer-visible outcomes — not infrastructure or domain internals. The specific metrics should reflect the product's business transactions, not generic traffic and error rates. - -Check: -- Does a Business group exist (named "Business", "B", or equivalent)? -- Is it the first group on the dashboard? -- Do its widgets use the `B0-N:` prefix? -- Does it contain 5-8 metrics covering: customer-visible success rates, key transaction flows, and SLA-impacting latency? -- Can someone determine "are customers affected?" within 5 seconds of opening the dashboard? - -**Findings format**: - -```markdown -#### Business Section Audit - -**Status**: MISSING / INCOMPLETE / OK - -**Current state**: [Description of what exists] - -**Recommended metrics** (if missing or incomplete): -1. B0: Key transaction success rate (are critical flows completing?) -2. B0: Customer-facing error rate (are requests failing for customers?) -3. B1: API p99 latency (are responses slow for customers?) -4. B1: Total request rate (are we receiving traffic?) -5. B2: Queue depth or processing lag (is async work backing up?) -6. B2: Key business event throughput (e.g. orders created, payments processed) -``` - -### 6. Apply Zero-Knowledge Viewer Test - -**Principle**: Someone with zero knowledge of the service should be able to spot problems by looking for red indicators. - -Evaluate: -- Can you identify a problem in under 10 seconds without reading widget titles? -- Are thresholds visible as red lines on every graph? -- Is conditional formatting applied to query value widgets (green/yellow/red)? -- Are group names self-explanatory? -- Is there a note widget with runbook links or team ownership? - -**Findings format**: - -```markdown -#### Zero-Knowledge Readability Audit - -| Check | Status | Finding | -|-------|--------|---------| -| Problems visible in <10s | FAIL | No red lines on 8 of 12 graphs | -| Conditional formatting on QV widgets | PARTIAL | 2 of 4 QV widgets have thresholds | -| Group names self-explanatory | OK | All groups use clear names | -| Runbook/ownership note | MISSING | No note widget with team info | -``` - -### 7. Generate Review Report - -Compile all findings into a structured report: - -```markdown -# Dashboard Review: [Dashboard Title] - -**Dashboard ID**: [id] -**URL**: [url] -**Review date**: [date] - -## Summary - -[2-3 sentence summary: overall health of the dashboard, critical issues count] - -## Critical Issues - -[List issues that must be fixed before the dashboard is production-ready] - -## Alert Threshold Audit -[From step 3] - -## Threshold Proximity Audit -[From step 4] - -## Business Section Audit -[From step 5] - -## Zero-Knowledge Readability Audit -[From step 6] - -## Recommended Actions - -### Must Fix -1. [Action item with specific widget and group reference] - -### Should Fix -1. [Action item] - -### Nice to Have -1. [Action item] -``` - ---- - -## Quality Checklist - -- [ ] Every widget title uses the layer-priority prefix (`I0:`, `P1:`, `D0:`, `B0:`, etc.) -- [ ] Every timeseries widget audited for alert threshold markers -- [ ] Threshold proximity checked (no large gaps between normal values and alert lines) -- [ ] Business group exists with 5-8 `B`-prefixed metrics at the top -- [ ] Zero-knowledge viewer test applied (red indicators visible without context) -- [ ] Query Value widgets checked for conditional formatting (green/yellow/red) -- [ ] All findings include specific widget names and group references -- [ ] Recommended actions categorized by priority (must/should/nice-to-have) -- [ ] Dashboard URL included in report for easy reference