[OTLP Validation] Run 26866372350 — Validation Report

[github-actions]

A. Executive Summary

Overall Status: PASS (with advisory notes)

Run: 26866372350 · github/gh-aw · main · schedule
Validation time: 2026-06-03T05:51:15Z – 2026-06-03T05:53:40Z
Service name: gh-aw.otlp-data-quality-validator
Trace ID: 7f25d80a01710b566d586a5f944c2c78

The single emitted span (gh-aw.agent.setup) is well-formed, carries all required attributes, and the trace ID is consistent between the JSONL mirror and the environment variable GITHUB_AW_OTEL_TRACE_ID. No export errors were recorded. Sentry network connectivity was confirmed via firewall logs (HTTP CONNECT 200). Two advisory items are noted:

1. No Grafana traffic observed in firewall logs — Grafana is configured as a second endpoint (GH_AW_OTLP_ENDPOINTS) but no connections to *.grafana.net appeared in the captured log window. The setup span export likely preceded the log capture window, so data loss is not confirmed. Confidence: medium.
2. JSONL mirror path mismatch — the workflow prompt and architecture diagram reference /tmp/gh-aw/agent/otel.jsonl, but the implementation writes to /tmp/gh-aw/otel.jsonl; error files are at /tmp/gh-aw/otlp-export-errors.* (not .../agent/). This is a documentation/spec drift that will cause confusion for future validators.

Conclusion and agent spans have not yet been emitted (workflow is still running) — no assessment of those span types is possible from this execution.

---

B. Trace Completeness

Item	Value
Spans in JSONL mirror	1 (gh-aw.agent.setup)
Expected spans at validation time	1 (setup only; conclusion/agent spans emitted after workflow finishes)
Missing spans	None (within current lifecycle phase)
Duplicate spans	0 (traceId+spanId unique: 1/1)
Unique trace IDs	1 ✅
Trace ID matches GITHUB_AW_OTEL_TRACE_ID	YES ✅
Export error count	0 (no otlp-export-errors.count file)

Confidence: HIGH for setup phase. Conclusion/agent span assessment deferred (workflow still running).

---

C. Span Hierarchy Validation

Check	Result	Detail
Root setup parent presence	✅ PASS	parentSpanId: a1be395e0b3a3dff (global activation parent)
Span naming pattern gh-aw.<job>.<op>	✅ PASS	gh-aw.agent.setup
kind = INTERNAL (1) for setup span	✅ PASS	kind: 1
Timestamp order (start ≤ end)	✅ PASS	start 05:51:15.817Z, end 05:51:17.817Z, duration 2000ms
Timestamps within run window	✅ PASS	within session (2026-06-03 05:51 UTC)
Conclusion spans parent under setup	⬜ DEFERRED	Not yet emitted
Agent spans parent under conclusion	⬜ DEFERRED	Not yet emitted

---

D. Attribute Contract Validation

Setup Span Required Attributes (spec §10.1)

Attribute	Present	Value
gh-aw.job.name	✅	"agent"
gh-aw.workflow.name	✅	"OTLP Data Quality Validator"
gh-aw.run.id	✅	"26866372350"
gh-aw.run.attempt	✅	"1"
gh-aw.run.actor	✅	"mnkiefer"
gh-aw.repository	✅	"github/gh-aw"
gh-aw.staged	✅	false

All 7 required setup span attributes present. ✅

Conditional attributes (present when applicable per §10.1):

• gen_ai.system: "github_models" ✅
• gh-aw.engine.id: "copilot" ✅
• gh-aw.event_name: "schedule" ✅

Conclusion Span Required Attributes (spec §10.2)

⬜ DEFERRED — span not yet emitted.

Agent Span GenAI Attributes (spec §10.3)

⬜ DEFERRED — span not yet emitted.

Resource Attributes (spec §11.1)

Attribute	Present	Value
service.name	✅	"gh-aw.otlp-data-quality-validator"
service.version	✅	"1.0.55"
github.repository	✅	"github/gh-aw"
github.run_id	✅	"26866372350"
github.run_attempt	✅	"1"
github.actions.run_url	✅	"https://github.com/github/gh-aw/actions/runs/26866372350"

All 6 required resource attributes present. ✅

Additional resource attributes observed (non-normative): github.event_name, github.ref, github.ref_name, github.sha, github.job, github.workflow_ref, github.actor_id, runner.os, runner.arch, runner.name, runner.environment, gh-aw.awf.version, deployment.environment, gh-aw.engine.id, gh-aw.workflow.name, gh-aw.repository, gh-aw.run.id.

Instrumentation Scope

Check	Result
scope.name = "gh-aw"	✅ PASS
scope.version = service.version ("1.0.55")	✅ PASS

---

E. Export and Fan-Out Health

Endpoint	Status	Evidence
Sentry (o205451.ingest.us.sentry.io)	✅ Network reachable	10+ HTTP CONNECT 200 tunnels in firewall log
Grafana (otlp-gateway-prod-eu-west-2.grafana.net)	⚠️ Not observed in log window	No *.grafana.net connections in captured firewall log
JSONL mirror (/tmp/gh-aw/otel.jsonl)	✅ Written	1 span line confirmed
Export error count	✅ 0	No otlp-export-errors.count file
Export error details	✅ 0	No otlp-export-errors.jsonl file
Sentry header rewrite (Authorization → x-sentry-auth)	✅ Applied	GH_AW_OTLP_ENDPOINTS shows x-sentry-auth=sentry sentry_key=...
Multi-endpoint fan-out independence	✅ Architecture correct	Concurrent fan-out via sendOTLPToAllEndpoints

Note on Grafana gap: The setup span was emitted at 05:51:15 UTC; the firewall access log begins at 05:52:11 UTC — a 56-second gap. OTLP export for the setup span likely completed before log capture began. The absence of Grafana entries does not confirm data loss. However, combined with the absence of any Grafana traffic in the full captured window (05:52:11–05:53:35), there is mild concern about whether Grafana exports are reaching the firewall. Zero export errors makes silent failure unlikely.

Backend visibility: No MCP tools are available to query Sentry or Grafana backends directly. Backend ingestion status is UNKNOWN — rely on export error files and firewall logs as proxies.

---

F. Root-Cause Hypothesis

Status: No failure to explain. All observable checks pass.

Advisory — Grafana traffic absence:

• Most likely: Export happened before firewall log capture window began (56s gap). Low severity.
• Alternative: Grafana endpoint secret was unresolved and if-missing: error triggered — but GH_AW_OTLP_ENDPOINTS contains a valid Grafana URL with a Base64 Basic auth token, so secrets resolved correctly.
• Alternative: Grafana endpoint returns HTTP 4xx/5xx without triggering export error logging — unlikely given the error recording logic in send_otlp_span.cjs.

Advisory — JSONL mirror path drift:

• Implementation writes to /tmp/gh-aw/otel.jsonl (confirmed).
• Workflow prompt and architecture comments reference /tmp/gh-aw/agent/otel.jsonl.
• This is documentation drift, not a runtime bug. Confidence: HIGH.

---

G. Recommended Fixes (Prioritized)

1. [ADVISORY] Update workflow prompt and architecture documentation to reflect the correct JSONL mirror path: /tmp/gh-aw/otel.jsonl and error files /tmp/gh-aw/otlp-export-errors.{count,jsonl}. The /tmp/gh-aw/agent/ prefix does not exist. This prevents future diagnostic confusion (spec §7).

2. [ADVISORY] Add a Grafana backend reachability check to this validator workflow. After span emission, attempt a lightweight query to otlp-gateway-prod-eu-west-2.grafana.net to confirm ingestion, providing cross-stage evidence beyond firewall proxy logs.

3. [FUTURE] Once conclusion and agent spans are emitted post-run, re-run this validator against the final otel.jsonl to validate spec §10.2 and §10.3 attributes and the full span hierarchy (spec §9.3).

---

H. Validation Queries Used

# Span summary from JSONL mirror (at /tmp/gh-aw/otel.jsonl)
jq -c '.resourceSpans[].scopeSpans[].spans[] | {name, traceId, spanId, parentSpanId, kind, status}' /tmp/gh-aw/otel.jsonl

# Setup span required attributes
jq '.resourceSpans[].scopeSpans[].spans[] | select(.name | endswith(".setup")) | [.attributes[].key] | sort' /tmp/gh-aw/otel.jsonl

# Resource attributes
jq -c '.resourceSpans[].resource.attributes[] | {(.key): .value}' /tmp/gh-aw/otel.jsonl | sort -u

# Instrumentation scope
jq -c '.resourceSpans[].scopeSpans[].scope' /tmp/gh-aw/otel.jsonl

# Trace ID consistency (JSONL vs env)
jq -r '.resourceSpans[].scopeSpans[].spans[].traceId' /tmp/gh-aw/otel.jsonl
echo $GITHUB_AW_OTEL_TRACE_ID

# Export errors
cat /tmp/gh-aw/otlp-export-errors.count 2>/dev/null || echo 0
cat /tmp/gh-aw/otlp-export-errors.jsonl 2>/dev/null

# Firewall OTLP traffic
grep -c 'sentry\|grafana' /tmp/gh-aw/sandbox/firewall/logs/access.log

---

Validated by OTLP Data Quality Validator workflow · run 26866372350 · gh-aw v1.0.55

Generated by 🧭 OTLP Data Quality Validator · sonnet46 2.1M · ◷

• expires on Jun 10, 2026, 5:58 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #36850

View newer issue

---

This action was performed automatically by the OTLP Data Quality Validator workflow.