feat: emit AI credits as OTEL span attributes by lpcox · Pull Request #4707 · github/gh-aw-firewall

lpcox · 2026-06-10T19:49:05Z

Problem

AI Credits (AIC) are not queryable in Sentry or Grafana. The api-proxy writes ai_credits_this_response and ai_credits_total to token-usage.jsonl, but never emits them as OTEL span attributes. This means:

has:gh-aw.aic returns 0 results in Sentry
span."gh-aw.aic" is absent from the Grafana/Tempo attribute index
AI credit values are not queryable or aggregatable in observability backends

See github/gh-aw#38424 for the full observability gap analysis.

Solution

Add setBudgetAttributes() to the OTEL module that emits per-request budget data as span attributes:

Attribute	Type	Description
`awf.ai_credits`	string	AI credits consumed by this request
`awf.ai_credits_total`	string	Running total AI credits for the session
`awf.model_units`	string	Effective (multiplier-adjusted) token units this request
`awf.model_units_total`	string	Running total effective token units for the session
`awf.model_multiplier`	string	Model cost multiplier applied

Called from the onUsage callback in upstream-response.js after computeTokenBudgetUsage() returns, so budget attributes land on the same span as gen_ai.usage.* token attributes.

Key Design Decisions

String types — emitted as strings (matching awf.cached_read, awf.cached_write, awf.reasoning) because Sentry silently drops unknown numeric custom attributes; Grafana/Tempo can still filter and display string values
No "token" in key names — effective-token counts use awf.model_units / awf.model_units_total to avoid Sentry's PII scrubbing rule that redacts values for keys containing "token"
Best-effort — wrapped in try/catch, never throws on error
No-op when disabled — returns immediately when OTEL is not enabled or budget is undefined
Attribute naming — uses awf. prefix (consistent with existing awf.cached_read, awf.reasoning)

Testing

4 new unit tests in otel.test.js covering: full budget, partial budget, undefined budget, null span
All 39 otel tests pass

Add setBudgetAttributes() to otel.js that emits per-request AI credits and effective token budget data as numeric span attributes: awf.ai_credits - AI credits consumed by this request awf.ai_credits_total - Running total AI credits for this session awf.effective_tokens - Effective tokens this request awf.effective_tokens_total - Running total effective tokens awf.model_multiplier - Model cost multiplier applied These are emitted as numeric values so Sentry and Grafana can aggregate them (sum/avg/p95) without the string-type blocking issue. Closes github/gh-aw#38424 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-10T19:50:40Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	96.44%	96.48%	📈 +0.04%
Statements	96.35%	96.39%	📈 +0.04%
Functions	98.77%	98.77%	➡️ +0.00%
Branches	90.74%	90.77%	📈 +0.03%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/config-writer.ts`	89.3% → 90.9% (+1.65%)	89.3% → 90.9% (+1.65%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

Adds OpenTelemetry span attributes for per-request AI-credits and effective-token budget data from the api-proxy, aiming to make these values queryable/aggregatable in observability backends (Sentry/Grafana/Tempo).

Changes:

Emit new awf.* budget span attributes via a new otel.setBudgetAttributes() helper.
Call setBudgetAttributes() from the HTTP upstream onUsage callback alongside existing setTokenAttributes().
Add unit tests for setBudgetAttributes() and extend the OTEL-disabled noop surface.

Show a summary per file

File	Description
containers/api-proxy/upstream-response.js	Calls `otel.setBudgetAttributes()` with the computed budget fields during usage tracking.
containers/api-proxy/proxy-request.js	Adds `setBudgetAttributes` to the OTEL-disabled noop shim to avoid missing-method calls.
containers/api-proxy/otel.test.js	Adds unit tests asserting new budget attributes are attached to spans.
containers/api-proxy/otel.js	Implements `setBudgetAttributes()` and exports it from the OTEL module.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 4/4 changed files
Comments generated: 3

+    if (budgetResult.ai_credits_this_response != null) {
+      attrs['awf.ai_credits'] = budgetResult.ai_credits_this_response;
+    }
+    if (budgetResult.ai_credits_total != null) {
+      attrs['awf.ai_credits_total'] = budgetResult.ai_credits_total;
+    }


+    if (budgetResult.effective_tokens_this_response != null) {
+      attrs['awf.effective_tokens'] = budgetResult.effective_tokens_this_response;
+    }
+    if (budgetResult.effective_tokens_total != null) {
+      attrs['awf.effective_tokens_total'] = budgetResult.effective_tokens_total;
+    }


lpcox · 2026-06-10T21:34:03Z

@copilot address the review feedback

…s to model_units

Copilot · 2026-06-10T21:37:35Z

@copilot address the review feedback

Addressed all three review items in commit fix(otel):

JSDoc — removed "resolves" so it no longer implies async: "Called from the onUsage callback after computeTokenBudgetUsage returns".
Sentry numeric drop — switched all new awf.* budget values to String(...), matching the established awf.cached_read/awf.cached_write/awf.reasoning pattern documented in docs/otel-sentry.md. Updated the doc's attribute table with the new fields.
"token" scrubbing — renamed awf.effective_tokens → awf.model_units and awf.effective_tokens_total → awf.model_units_total. Updated tests accordingly; all 39 otel tests pass.

github-actions · 2026-06-10T22:02:25Z

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit 4712d2c

github-actions · 2026-06-10T22:07:16Z

🔥 Smoke Test: PAT Auth — PASS

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP connectivity	✅
File write/read	✅ `smoke-test-copilot-pat-27308006805.txt`

PR: feat: emit AI credits as OTEL span attributes
Author: @lpcox · No assignees
Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Overall: PASS

Note

🔒 Integrity filter blocked 1 item

The following item was blocked because it doesn't meet the GitHub integrity level.

#4706 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

🔑 PAT report filed by Smoke Copilot PAT

github-actions · 2026-06-10T22:07:31Z

🔥 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Status	Details
Module Loading	✅	`otel.js` loads; exports: `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled`
Test Suite	✅	39/39 tests pass (`otel.test.js`)
Env Var Forwarding	✅	`api-proxy-service-config.ts` forwards `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `GITHUB_AW_OTEL_TRACE_ID`, `GITHUB_AW_OTEL_PARENT_SPAN_ID`, `OTEL_SERVICE_NAME`; `observability-environment.ts` auto-forwards all `OTEL_*` vars
Token Tracker Integration	✅	`onUsage` callback present in `token-tracker-http.js`; `upstream-response.js` calls `otel.setBudgetAttributes(span, budgetResult)` via the hook
OTEL Diagnostics	➖	No live container run; graceful degradation confirmed (no errors when `OTEL_EXPORTER_OTLP_ENDPOINT` is unset — falls back to local file at `/var/log/api-proxy/otel.jsonl`)

All scenarios pass ✅

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-06-10T22:07:47Z

@lpcox

Smoke Test Results:

GitHub MCP API: ✅
GitHub.com HTTP: ✅
Agent file I/O: ✅
Direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw): ✅

Overall: PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-06-10T22:07:52Z

✅ fix(cli-proxy): resolve IPv4/IPv6 readiness probe mismatch on dual-stack hosts
✅ fix: propagate config fields to all layers
✅ GitHub PR queries
✅ File write/read
✅ Discussion lookup + comment
✅ GitHub homepage title check
✅ npm ci && npm run build
Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-06-10T22:08:02Z

🔍 Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	`Python 3.12.13`	`Python 3.12.3`	❌ NO
Node.js	`v24.16.0`	`v22.22.3`	❌ NO
Go	`go1.22.12`	`go1.22.12`	✅ YES

Overall: ❌ FAILED — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

github-actions · 2026-06-10T22:08:08Z

GitHub API: ✅ PASS
GitHub check: ✅ PASS
File verify: ✅ PASS

Total: PASS

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-06-10T22:08:09Z

🔬 Smoke Test Results — PR #4707

Test	Result
GitHub MCP connectivity	✅ `list_pull_requests` succeeded
GitHub.com HTTP	✅ HTTP 200
File write/read	❌ Pre-step template vars unresolved (`${{ steps.smoke-data.outputs.* }}`)

Overall: FAIL — file test unverifiable due to workflow variable injection failure.

PR by @lpcox · No assignees.

Note

🔒 Integrity filter blocked 1 item

The following item was blocked because it doesn't meet the GitHub integrity level.

#4706 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-06-10T22:08:47Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #4707 · 224.4 AIC · ⊞ 33.8K · ◷

github-actions · 2026-06-10T22:09:04Z

@lpcox @${{ github.event.pull_request.assignees }}

Smoke Test Results:

🔄 PR titles: feat: emit AI credits as OTEL span attributes
✅ GitHub MCP connectivity
✅ GitHub.com HTTP 200/301
✅ File write/read test
✅ Direct BYOK mode test

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

api.openai.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.openai.com"

See Network Configuration for more information.

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-06-10T22:09:54Z

Smoke Test Results — FAIL

host.docker.internal resolves to 172.17.0.1 but services are not reachable.

Check	Result
Redis PING	❌ timeout
PostgreSQL pg_isready	❌ no response
PostgreSQL SELECT 1	❌ timeout

Overall: FAIL — service containers are not accessible from this runner environment.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-06-10T22:17:53Z

Smoke Test: Copilot BYOK (Direct) Mode ✅ PASS

Test Results:

✅ GitHub MCP connectivity: 2 recent closed PRs verified
✅ GitHub.com connectivity: HTTP 200
✅ File write/read: /tmp/gh-aw/agent/smoke-test-copilot-byok.txt exists
✅ BYOK inference: Direct mode via api-proxy → api.githubcopilot.com working

Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) with api-proxy sidecar
Author: @lpcox

🔑 BYOK report filed by Smoke Copilot BYOK

lpcox · 2026-06-10T22:28:07Z

@copilot address review feedback

Copilot · 2026-06-10T22:29:36Z

@copilot address review feedback

Both open review threads were already addressed in commit d2e2a53:

Numeric attrs — all awf.* budget values are emitted as String(...) (matching awf.cached_read/awf.cached_write/awf.reasoning)
"token" in key names — awf.effective_tokens / awf.effective_tokens_total were renamed to awf.model_units / awf.model_units_total

The review threads reference the original commit e7942eb; the fixes are live. You can resolve both threads.

Copilot AI review requested due to automatic review settings June 10, 2026 19:49

Copilot started reviewing on behalf of lpcox June 10, 2026 19:49 View session

lpcox temporarily deployed to aoai-model June 10, 2026 19:50 — with GitHub Actions Inactive

Copilot AI reviewed Jun 10, 2026

View reviewed changes