Add smoke-token-telemetry workflow to catch token telemetry emitter regressions by Copilot · Pull Request #43403 · github/gh-aw

Copilot · 2026-07-04T16:05:13Z

Token telemetry (token_usage.jsonl / run_summary.json) was silently broken for 13+ days before detection, blinding all downstream token-cost and api-consumption audits. There was no CI guard to catch regressions early.

Changes

New workflow: `smoke-token-telemetry.md`

Daily schedule + push-to-main trigger; uses the real copilot engine (no engine.command override) so the AWF firewall proxy actually emits token_usage.jsonl
Adds a check_token_telemetry custom job (needs: [agent]) that downloads the agent artifact after a successful run and asserts:
1. At least one token_usage.jsonl is non-empty across the three sandbox paths
2. agent_usage.json reports input_tokens > 0
Fails the workflow immediately on violation with ::error:: annotations referencing the tracking issue

check_token_telemetry:
  needs: agent
  if: needs.agent.result == 'success'
  steps:
    - Assert token_usage.jsonl is non-empty   # scans all 3 sandbox proxy log paths
    - Assert agent_usage.json has non-zero token counts

Updated: `docs/src/content/docs/agent-factory-status.mdx`

Added the new workflow to the status table.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…-empty Closes #43217 Adds a new smoke CI workflow (.github/workflows/smoke-token-telemetry.md) that runs a minimal copilot LLM call daily and asserts: - At least one token_usage.jsonl file in the agent artifact is non-empty - agent_usage.json reports non-zero input_tokens This catches token-telemetry emitter regressions in hours rather than weeks, protecting api-consumption, detection, and cost audits. The check_token_telemetry custom job runs after the agent job and before the conclusion job; the overall workflow fails fast if the AWF firewall proxy token emitter is broken. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-07-04T18:46:55Z

🤖 PR Triage — Run §28715668077

Field	Value
Category	`test`
Risk	🟢 Low
Score	48/100
Impact	24/50
Urgency	14/30
Quality	10/20
Action	`batch_review`
Batch	`pr-batch:new-workflows`

Adds smoke-token-telemetry workflow. +98 md, +1784 lock.yml, +1 docs. Draft, no CI yet.

Generated by 🔧 PR Triage Agent · 113.5 AIC · ⌖ 13 AIC · ⊞ 5.5K · ◷

pelikhan · 2026-07-05T03:21:07Z

@copilot refactor job into a shared agentic workflow and import into smoke-copilot, smoke-Claude, smoke-codex. Remove new smoke-telemetry agentic workflow

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…import into smoke-copilot, smoke-claude, smoke-codex; remove standalone smoke-token-telemetry workflow Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-05T03:31:10Z

@copilot refactor job into a shared agentic workflow and import into smoke-copilot, smoke-Claude, smoke-codex. Remove new smoke-telemetry agentic workflow

Done in the latest commit. The check_token_telemetry job is now extracted into shared/token-telemetry-check.md and imported into smoke-copilot.md, smoke-claude.md, and smoke-codex.md. The standalone smoke-token-telemetry workflow and its lock file have been removed.

pelikhan · 2026-07-05T03:33:09Z

@copilot use download artifact v8

Copilot · 2026-07-05T03:47:40Z

@copilot use download artifact v8

shared/token-telemetry-check.md already uses actions/download-artifact@v8.0.1.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-05T03:49:43Z

@copilot import the share agentic workflow to 10 more agentic workflows in this repository

Copilot

Pull request overview

This PR introduces a shared check_token_telemetry job intended to run after smoke workflows’ agent job and validate that firewall token telemetry artifacts are being emitted (via token-usage.jsonl and agent_usage.json) so regressions are caught by CI.

Changes:

Add a shared workflow fragment (shared/token-telemetry-check.md) defining check_token_telemetry.
Import that shared check into existing smoke workflows (copilot/codex/claude) and update their compiled .lock.yml outputs to include the new job.
Update the repository’s Actions lock metadata (.github/aw/actions-lock.json) as part of the workflow compilation/locking process.

Show a summary per file

File	Description
.github/workflows/smoke-copilot.md	Imports the shared token telemetry check into the copilot smoke workflow.
.github/workflows/smoke-copilot.lock.yml	Compiled workflow updated to include the `check_token_telemetry` job and wire it into `conclusion`.
.github/workflows/smoke-codex.md	Imports the shared token telemetry check into the codex smoke workflow.
.github/workflows/smoke-codex.lock.yml	Compiled workflow updated to include the `check_token_telemetry` job and wire it into `conclusion`.
.github/workflows/smoke-claude.md	Imports the shared token telemetry check into the claude smoke workflow.
.github/workflows/smoke-claude.lock.yml	Compiled workflow updated to include the `check_token_telemetry` job and wire it into `conclusion`.
.github/workflows/shared/token-telemetry-check.md	New shared job definition that downloads the agent artifact and asserts token telemetry outputs exist.
.github/aw/actions-lock.json	Regenerated/normalized Actions lock data consistent with the updated workflows.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 7/8 changed files
Comments generated: 4
Review effort level: Low

+    permissions:
+      contents: read


+      - name: Download agent artifact
+        id: download-agent
+        continue-on-error: true
+        uses: actions/download-artifact@v8.0.1
+        with:
+          name: agent
+          path: /tmp/gh-aw/
+


+            echo "::error::All token_usage.jsonl files are empty or missing after a successful agent run."
+            echo "::error::The AWF firewall proxy token telemetry emitter may be broken."
+            echo "::error::See tracking issue: https://github.com/github/gh-aw/issues/42791"
+            exit 1


  - shared/github-queries-mcp-script.md
  - shared/mcp/serena-go.md
  - shared/otlp.md
+  - shared/token-telemetry-check.md


gh-aw-bot · 2026-07-05T03:56:32Z

@copilot please run the pr-finisher skill, address the latest maintainer feedback by importing the shared token-telemetry workflow into 10 more relevant agentic workflows in this repository, then recompile affected workflows and rerun checks.

Generated by 👨‍🍳 PR Sous Chef · 4.83 AIC · ⌖ 6.02 AIC · ⊞ 3.4K · ◷
_{Comment /souschef to run again}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-05T04:03:27Z

@copilot import the share agentic workflow to 10 more agentic workflows in this repository

Implemented in c693971: imported shared/token-telemetry-check.md into 10 additional smoke workflows (smoke-gemini, smoke-opencode, smoke-copilot-arm, smoke-copilot-aoai-apikey, smoke-copilot-aoai-entra, smoke-temporary-id, smoke-test-tools, smoke-crush, smoke-service-ports, smoke-project) and recompiled their .lock.yml files.

github-actions · 2026-07-07T10:08:24Z

🎉 This pull request is included in a new release.

Release: v0.82.3

Initial plan

4e278d4

Copilot AI assigned Copilot and pelikhan Jul 4, 2026

Copilot started work on behalf of pelikhan July 4, 2026 16:05 View session

Copilot AI linked an issue Jul 4, 2026 that may be closed by this pull request

[deep-report] Add CI smoke assertion that token_usage.jsonl is non-empty for agent runs #43217

Closed

Copilot AI and others added 2 commits July 4, 2026 16:19

Initial plan setup

028ed7a

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add CI smoke assertion for non-empty token_usage.jsonl~~ Add smoke-token-telemetry workflow to catch token telemetry emitter regressions Jul 4, 2026

Copilot finished work on behalf of pelikhan July 4, 2026 16:27

Copilot AI requested a review from pelikhan July 4, 2026 16:27

github-actions Bot added pr-action:batch_review pr-agent:copilot-swe-agent pr-batch:new-workflows pr-priority:medium pr-risk:low pr-type:test labels Jul 4, 2026

This was referenced Jul 4, 2026

[PR Triage Report] 🤖 PR Triage Report — 2026-07-04 (Run §28715668077) #43427

Closed

[PR Triage Report] PR Triage Report — 2026-07-05 (Run §28725134784) #43445

Closed

Copilot started work on behalf of pelikhan July 5, 2026 03:21 View session

Copilot AI and others added 2 commits July 5, 2026 03:25

Initial plan: refactor check_token_telemetry into shared workflow

580666b

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Refactor check_token_telemetry into shared/token-telemetry-check.md; …

21c496f

…import into smoke-copilot, smoke-claude, smoke-codex; remove standalone smoke-token-telemetry workflow Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 5, 2026 03:31

Copilot started work on behalf of pelikhan July 5, 2026 03:43 View session

Apply remaining changes

03a0c92

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 5, 2026 03:48

pelikhan marked this pull request as ready for review July 5, 2026 03:48

Copilot AI review requested due to automatic review settings July 5, 2026 03:48

Copilot started reviewing on behalf of pelikhan July 5, 2026 03:49 View session

Copilot AI reviewed Jul 5, 2026

View reviewed changes

Copilot started work on behalf of pelikhan July 5, 2026 03:52 View session

Import shared token telemetry check into 10 more smoke workflows

c693971

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan closed this Jul 5, 2026

Copilot stopped work on behalf of pelikhan due to an error July 5, 2026 04:07
The session was cancelled by the user.

pelikhan reopened this Jul 5, 2026

pelikhan merged commit e40c8df into main Jul 5, 2026

pelikhan deleted the copilot/deep-report-add-ci-smoke-assertion branch July 5, 2026 04:08

github-actions Bot mentioned this pull request Jul 5, 2026

[PR Triage Report] PR Triage Report — Run #7 (2026-07-05) #43502

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add smoke-token-telemetry workflow to catch token telemetry emitter regressions#43403

Add smoke-token-telemetry workflow to catch token telemetry emitter regressions#43403
pelikhan merged 7 commits into
mainfrom
copilot/deep-report-add-ci-smoke-assertion

Copilot AI commented Jul 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 4, 2026

Uh oh!

pelikhan commented Jul 5, 2026

Uh oh!

Copilot AI commented Jul 5, 2026

Uh oh!

pelikhan commented Jul 5, 2026

Uh oh!

Copilot AI commented Jul 5, 2026

Uh oh!

pelikhan commented Jul 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gh-aw-bot commented Jul 5, 2026

Uh oh!

Copilot AI commented Jul 5, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Copilot AI commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

New workflow: smoke-token-telemetry.md

Updated: docs/src/content/docs/agent-factory-status.mdx

Uh oh!

github-actions Bot commented Jul 4, 2026

🤖 PR Triage — Run §28715668077

Uh oh!

pelikhan commented Jul 5, 2026

Uh oh!

Copilot AI commented Jul 5, 2026

Uh oh!

pelikhan commented Jul 5, 2026

Uh oh!

Copilot AI commented Jul 5, 2026

Uh oh!

pelikhan commented Jul 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

gh-aw-bot commented Jul 5, 2026

Uh oh!

Copilot AI commented Jul 5, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 4, 2026 •

edited

Loading

New workflow: `smoke-token-telemetry.md`

Updated: `docs/src/content/docs/agent-factory-status.mdx`