refactor: split pkg/cli/token_usage.go (1141 lines) into 6 focused modules

docs/adr/48455-split-token-usage-monolith-into-focused-modules.md

@@ -0,0 +1,48 @@
+# ADR-48455: Split token_usage.go Monolith into Focused Modules
+
+**Date**: 2026-07-28
+**Status**: Draft
+**Deciders**: Unknown (automated refactor by copilot-swe-agent)
+
+---
+
+### Context
+
+`pkg/cli/token_usage.go` had grown to 1,141 lines by mixing unrelated concerns in a single file: data type definitions, file discovery logic, JSONL parsing, token usage analysis, API proxy steering event counting, and sub-agent model attribution. This made the file hard to navigate, understand, and test in isolation. The Go convention of grouping code into focused, single-responsibility files was not being followed, and the growing complexity increased the risk of unintended coupling between these concerns.
+
+### Decision
+
+We will split `pkg/cli/token_usage.go` into six focused modules, each owning a single responsibility: `token_usage_types.go` (structs/constants), `token_usage_find.go` (file discovery), `token_usage_parse.go` (JSONL parsing), `token_usage_analyze.go` (aggregation logic), `token_usage_steering.go` (steering event counting), and `token_usage_subagent.go` (sub-agent attribution). No logic changes, no API surface changes — this is a pure structural reorganization within the same `cli` package.
+
+### Alternatives Considered
+
+#### Alternative 1: Keep the monolithic file
+
+Leave `token_usage.go` as-is and continue adding to it. This avoids all churn and merge risk, but the file will continue to grow, making navigation and testing harder over time. Rejected because the 1,141-line size already exceeds a navigable threshold and no natural stopping point exists.
+
+#### Alternative 2: Extract to a dedicated sub-package (`pkg/cli/tokenusage/`)
+
+Move all token usage logic into its own sub-package. This would provide stronger encapsulation via Go's package visibility rules. Rejected for this PR because it would change import paths, affect the public API surface, and is a larger structural change that should be a separate, explicit decision with its own ADR. The within-package file split is a lower-risk first step.
+
+#### Alternative 3: Extract only types to a shared package
+
+Move `TokenUsageSummary` and related types to a shared `pkg/tokenusage` package to break potential import cycles elsewhere. Rejected because no import cycles currently exist, and the types are only consumed within the `cli` package; moving them would add premature abstraction.
+
+### Consequences
+
+#### Positive
+- Each file is now under ~400 lines and has a clear, single responsibility, reducing cognitive load when navigating or modifying token usage logic.
+- Focused files make it easier to write targeted unit tests for individual concerns (e.g., testing file discovery independently of parsing).
+- The split establishes a clear convention for future growth: new concerns get their own file rather than appending to a monolith.
+
+#### Negative
+- The number of files in `pkg/cli/` increases by five, which may make directory listings noisier.
+- Future refactors that need to move token usage logic to its own sub-package will still need to happen; this split does not resolve the underlying question of whether this logic belongs in `pkg/cli/` long-term.
+
+#### Neutral
+- All existing tests pass unchanged — the split is transparent to callers and tests.
+- The `pkg/cli` package boundary is preserved; no import path changes are required.
+
+---
+
+*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

docs/src/content/docs/specs/effective-tokens-specification.md

@@ -887,17 +887,17 @@ The table below maps the normative sections of this specification to the impleme
 | §5 Multi-Invocation Aggregation | `ET_total`, `raw_total_tokens`, `total_invocations` | `pkg/cli/effective_tokens.go` (`AggregateEffectiveTokens`) |
 | §6 Execution Graph Requirements | Node schema, root/sub-agent linkage, graph traversal | `pkg/cli/logs_models.go`, `pkg/cli/logs_episode.go`, `pkg/cli/logs_orchestrator.go` |
 | §7 Reporting | Console and JSON output of ET summaries and per-model breakdowns | `pkg/cli/audit_report.go`, `pkg/cli/audit_report_render_tools.go`, `pkg/cli/audit_diff.go`, `pkg/cli/logs_report.go` |
-| §7.1 OTel Attribute Requirements | OpenTelemetry span attribute emission for ET metrics | `pkg/cli/token_usage.go`, `pkg/cli/logs_run_processor.go` |
+| §7.1 OTel Attribute Requirements | OpenTelemetry span attribute emission for ET metrics | `pkg/cli/token_usage_types.go`, `pkg/cli/token_usage_parse.go`, `pkg/cli/logs_run_processor.go` |
 | §8 Implementation Requirements | Completeness, determinism, versioning, partial visibility safeguards | `pkg/cli/effective_tokens.go`, `pkg/cli/forecast_montecarlo.go` |
 
 ### §7.1 OTel Attribute Row-to-Code Mapping
 
 | §7.1 Attribute Key | Implementation Mapping |
 |---|---|
-| `llm.token.effective_total` | `pkg/cli/token_usage.go` → `TokenUsageSummary.TotalEffectiveTokens`, populated by `populateEffectiveTokensWithCustomWeights` |
-| `llm.token.input` | `pkg/cli/token_usage.go` → `TokenUsageEntry.InputTokens` and `ModelTokenUsage.InputTokens`, aggregated in `parseTokenUsageFile` |
-| `llm.token.output` | `pkg/cli/token_usage.go` → `TokenUsageEntry.OutputTokens` and `ModelTokenUsage.OutputTokens`, aggregated in `parseTokenUsageFile` |
-| `llm.token.cached_input` | `pkg/cli/token_usage.go` → `TokenUsageEntry.CacheReadTokens` and `ModelTokenUsage.CacheReadTokens`, aggregated in `parseTokenUsageFile` |
+| `llm.token.effective_total` | `pkg/cli/token_usage_types.go` → `TokenUsageSummary.TotalEffectiveTokens`, populated by `populateEffectiveTokensWithCustomWeights` |
+| `llm.token.input` | `pkg/cli/token_usage_parse.go` + `pkg/cli/token_usage_types.go` → `TokenUsageEntry.InputTokens` and `ModelTokenUsage.InputTokens`, aggregated in `parseTokenUsageFile` |
+| `llm.token.output` | `pkg/cli/token_usage_parse.go` + `pkg/cli/token_usage_types.go` → `TokenUsageEntry.OutputTokens` and `ModelTokenUsage.OutputTokens`, aggregated in `parseTokenUsageFile` |
+| `llm.token.cached_input` | `pkg/cli/token_usage_parse.go` + `pkg/cli/token_usage_types.go` → `TokenUsageEntry.CacheReadTokens` and `ModelTokenUsage.CacheReadTokens`, aggregated in `parseTokenUsageFile` |
 | `llm.token.base_weighted` | `pkg/cli/effective_tokens.go` → base token weighting in `computeModelEffectiveTokensWithWeights` (pre-multiplier term) |
 | `llm.model.multiplier` | `pkg/cli/effective_tokens.go` → multiplier resolution in `computeModelEffectiveTokensWithWeights` (`mult` selection by model key/prefix) |
 
@@ -909,7 +909,7 @@ To keep the specification and implementation synchronized:
 2. When changing aggregation semantics (§5), update `pkg/cli/effective_tokens.go` and rerun tests `T-ET-010–T-ET-012` and `T-ET-006`.
 3. When changing the execution graph node schema (§6), update `pkg/cli/logs_models.go` and `pkg/cli/logs_episode.go` in the same change.
 4. When changing reporting format or field names (§7), update the affected render files in `pkg/cli/` and run `go test ./pkg/cli/ -run TestAudit`.
-5. When changing OTel attribute names (§7.1), update `pkg/cli/token_usage.go` and verify attribute names with `grep -r "effective_tokens" pkg/`.
+5. When changing OTel attribute names (§7.1), update `pkg/cli/token_usage_types.go` and `pkg/cli/token_usage_parse.go` and verify attribute names with `grep -r "effective_tokens" pkg/`.
 6. After any §8 change affecting determinism or partial visibility, re-run `go test ./pkg/cli/ -run TestEffectiveTokens` and `go test ./pkg/cli/ -run TestRunMonteCarlo`.
 
 Run `grep -r "effective_tokens" pkg/` to confirm all implementation files are captured in the table above.

pkg/cli/README.md

@@ -645,8 +645,8 @@ This appendix is generated from the current non-test Go source files in this pac
 | `outcome_evaluation.go` | `OutcomeStatus` | `type OutcomeStatus string` | OutcomeStatus is the normalized classification for a safe output outcome. |
 | `packages.go` | `IncludeDependency` | `type IncludeDependency struct { SourcePath string // Path in the source (local) TargetPath string // Relative path where it should be copied in .github/workflows IsOptional bool // Whether this is an optional include (@include?) }` | IncludeDependency represents a file dependency from @include directives |
 | `run_interactive.go` | `RunWorkflowOptions` | `type RunWorkflowOptions struct { WorkflowName string Verbose bool EngineOverride string RepoOverride string RefOverride string AutoMergePRs bool Push bool DryRun bool }` | RunWorkflowOptions holds parameters for RunSpecificWorkflowInteractively. |
-| `token_usage.go` | `SubagentModelActual` | `type SubagentModelActual struct { Model string `json:"model"` Provider string `json:"provider,omitempty"` Requests int `json:"requests"` }` | SubagentModelActual captures model usage observed in token-usage logs. |
-| `token_usage.go` | `SubagentModelRequest` | `type SubagentModelRequest struct { AgentName string `json:"agent_name"` RequestedModel string `json:"requested_model"` InvocationCount int `json:"invocation_count"` EffectiveModel string `json:"effective_model,omitempty"` ReasonCode string `json:"reason_code,omitempty"` }` | SubagentModelRequest captures requested/effective model attribution for a sub-agent. |
+| `token_usage_types.go` | `SubagentModelActual` | `type SubagentModelActual struct { Model string `json:"model"` Provider string `json:"provider,omitempty"` Requests int `json:"requests"` }` | SubagentModelActual captures model usage observed in token-usage logs. |
+| `token_usage_types.go` | `SubagentModelRequest` | `type SubagentModelRequest struct { AgentName string `json:"agent_name"` RequestedModel string `json:"requested_model"` InvocationCount int `json:"invocation_count"` EffectiveModel string `json:"effective_model,omitempty"` ReasonCode string `json:"reason_code,omitempty"` }` | SubagentModelRequest captures requested/effective model attribution for a sub-agent. |
 | `update_workflows.go` | `UpdateWorkflowsOptions` | `type UpdateWorkflowsOptions struct { WorkflowNames []string AllowMajor bool Force bool Yes bool Verbose bool EngineOverride string WorkflowsDir string NoStopAfter bool StopAfter string NoMerge bool DisableReleaseBump bool DisableSecurityScanner bool NoCompile bool NoRedirect bool CoolDown time.Duration }` | UpdateWorkflowsOptions configures workflow update behavior. |
 | `view_command.go` | `ViewOptions` | `type ViewOptions struct { Owner string Repo string Hostname string OutputDir string Verbose bool }` | ViewOptions holds configuration for the view command. |
 
@@ -758,9 +758,9 @@ This appendix is generated from the current non-test Go source files in this pac
 | `packages.go` | `ExtractWorkflowPrivateSetting` | `func ExtractWorkflowPrivateSetting(content string) (bool, bool)` | ExtractWorkflowPrivateSetting extracts the private field from workflow content string. |
 | `pr_automerge.go` | `AutoMergePullRequestsLegacy` | `func AutoMergePullRequestsLegacy(repoSlug string, verbose bool) error` | AutoMergePullRequestsLegacy is the legacy function that auto-merges all open PRs (used by trial command for backward compatibility) |
 | `project_timezone.go` | `ConfigureProjectTimezone` | `func ConfigureProjectTimezone()` | ConfigureProjectTimezone applies the configured project timezone to CLI time rendering. |
-| `token_usage.go` | `(*TokenUsageSummary).AvgDurationMs` | `func (*TokenUsageSummary).AvgDurationMs() int` | AvgDurationMs returns the average request duration in milliseconds |
-| `token_usage.go` | `(*TokenUsageSummary).ModelRows` | `func (*TokenUsageSummary).ModelRows() []ModelTokenUsageRow` | ModelRows returns the by-model data as sorted rows for console rendering |
-| `token_usage.go` | `(*TokenUsageSummary).TotalTokens` | `func (*TokenUsageSummary).TotalTokens() int` | TotalTokens returns the sum of all token types |
+| `token_usage_analyze.go` | `(*TokenUsageSummary).AvgDurationMs` | `func (*TokenUsageSummary).AvgDurationMs() int` | AvgDurationMs returns the average request duration in milliseconds |
+| `token_usage_analyze.go` | `(*TokenUsageSummary).ModelRows` | `func (*TokenUsageSummary).ModelRows() []ModelTokenUsageRow` | ModelRows returns the by-model data as sorted rows for console rendering |
+| `token_usage_analyze.go` | `(*TokenUsageSummary).TotalTokens` | `func (*TokenUsageSummary).TotalTokens() int` | TotalTokens returns the sum of all token types |
 | `tool_graph.go` | `(*ToolGraph).AddSequence` | `func (*ToolGraph).AddSequence(tools []string)` | AddSequence adds a tool call sequence to the graph |
 | `tool_graph.go` | `(*ToolGraph).GenerateMermaidGraph` | `func (*ToolGraph).GenerateMermaidGraph() string` | GenerateMermaidGraph generates a Mermaid state diagram from the tool graph |
 | `tool_graph.go` | `NewToolGraph` | `func NewToolGraph() *ToolGraph` | NewToolGraph creates a new empty tool graph |

pkg/cli/gateway_logs_mcp.go

@@ -263,3 +263,9 @@ func buildMCPSummaryStats(gatewayMetrics *GatewayMetrics, mcpData *MCPToolUsageD
 		}
 	})
 }
+
+// TODO: Implement token-usage correlation for MCP tool calls.
+func correlateToolCallsWithTokenDelta(toolCalls []MCPToolCall, tokenUsageFile string) []MCPToolCall {
+	_ = tokenUsageFile
+	return toolCalls
+}