feat(explain): live progress and observable-state timeout diagnostic

[peyton-alt]

https://entire.io/gh/entireio/cli/trails/164

Summary

Two paired user-facing improvements to entire explain --generate, building on PR #1204 (which made the 30s timeout configurable but kept a 5-minute default and didn't change the silent-wait UX):

1. Live progress on TTY. Real-time phase events (sending request → first token / TTFT → writing tokens → done) instead of silent wait. In-place line updates on TTY; one line per event in CI/piped; ASCII glyphs under ACCESSIBLE=1.
2. Observable-state timeout diagnostic. When --summary-timeout-seconds fires, the error message reflects what the subprocess was actually doing. Streaming providers (Claude) get phase-specific attribution; non-streaming providers (Codex/Cursor/Copilot/Gemini) surface captured stderr and distinguish "produced no output" from "was generating output when killed". The three-cause shrug from PR Add timeout to explain summary generation #876 is removed.

Plus one policy change that enables both:

3. Automatic deadline removed. --generate no longer has a 5-minute default cap. Timeouts are opt-in only via --summary-timeout-seconds flag or summary_timeout_seconds setting (both surfaces from Honor summary_timeout_seconds in explain --generate; raise default to 5m #1204 preserved). With no automatic cap, every diagnostic users see is for a timeout they consciously requested.

What users see now

TTY, normal run:

Generating checkpoint summary... (transcript: 40.7 KB, provider: claude-code)
→ Sending request to provider...
→ Provider responded (TTFT 1.6s, 13k cached input tokens) -- generating...
→ Writing summary... (~Nk tokens)             ← updates in place
✓ Summary generated (7.6s, 420 output tokens)
✓ Summary generated for <checkpoint-id>

TTY, network down + --summary-timeout-seconds 10:

Generating checkpoint summary... (transcript: 40.7 KB, provider: claude-code)
→ Sending request to provider...
✗ Timed out after 10s: provider sent request but received no response
  cause    network/firewall, provider API degraded, or auth check stuck
  try      check connectivity to the provider, then retry

TTY, transcript too large + --summary-timeout-seconds 3:

→ Provider responded (TTFT 1.6s, 20.2k cached input tokens) -- generating...
→ Writing summary... (~24 tokens)
✗ Timed out after 3s: model responded but did not finish
  cause    transcript may be too large for the chosen cap, or model is slow
  try      raise --summary-timeout-seconds or pick a faster model

Non-TTY (| cat): same lines, one per event, no \r in-place updates, no ANSI styling (lipgloss auto-strips).

ACCESSIBLE=1: → becomes ->, no ANSI escapes.

Architecture

• agent.StreamingTextGenerator optional capability interface with AsStreamingTextGenerator detector. Defined alongside the existing TextGenerator interface. Claude Code is the first (currently only) implementer.
• (*ClaudeCodeAgent).GenerateTextStreaming invokes claude --print --output-format stream-json --include-partial-messages --verbose, pipes stdout through a 4 MiB-buffered NDJSON scanner, and dispatches ProgressPhase callbacks (Connecting → FirstToken → Generating(×N) → Done). Falls back to plain GenerateText when stderr indicates the CLI rejected streaming flags (older Claude CLI builds).
• summarize.TextGeneratorAdapter (the production summary-generator path used by all providers via resolveCheckpointSummaryProvider) gained streaming-capability detection and dispatches to GenerateTextStreaming when available. summarize.ClaudeGenerator got the same treatment for legacy callers.
• summaryProgressWriter renders progress events using existing statusStyles (lipgloss). In-place \r updates on TTY; one line per event otherwise; ASCII glyphs in accessible mode. Throttled to 500ms-OR-25%-jump in non-TTY mode to avoid spam.
• summaryAttempt value type captures observable state during a run (phases reached, captured stderr, stdout byte count) so the diagnostic path can attribute timeouts without depending on the UI struct.
• *agent.TextGenerationError wraps RunIsolatedTextGeneratorCLI failures with captured stderr + stdout byte count, so the diagnostic can surface real subprocess output for non-streaming providers.
• providerCLIName maps types.AgentName to the user-facing CLI binary in diagnostic try: rows (run \claude` directly to confirm it works`).

What changed from the original draft design (in case you remember the old PR)

The original draft (commits before this rebase) had a TTY-gated deadline strategy with an idle-based 5min watchdog + 30min hard wall-clock cap, plus an opt-in summary_timeout_seconds override. This redesign drops the automatic deadline entirely — timeouts are pure opt-in via the flag/setting from #1204. Rationale: the bug behind issue #1198 was that the auto-default truncated legitimate generations. Making the deadline opt-in fixes the root cause and removes a layer of mechanism (no idle watchdog, no separate wall cap, no goroutine).

Also intentionally NOT included: a stderr substring classifier for non-streaming providers (would have been "chunk 8" in the original plan). Decision: ship verbatim stderr passthrough rather than speculative phrase-matching without real CLI fixtures. The raw stderr is always surfaced in the diagnostic rows.

Test plan

• mise run check (fmt + lint + unit + integration + E2E Vogon canary) passes locally
• Manual: TTY happy path — progress phases render with in-place updates, final ✓ with duration + token count
• Manual: --summary-timeout-seconds 3 on a real Claude generation → "model responded but did not finish" with cause + try rows
• Manual: Network offline + --summary-timeout-seconds 10 → "provider sent request but received no response" (phase-distinguished from "never sent its request")
• Manual: | cat → one line per event, no \r, no ANSI
• Manual: ACCESSIBLE=1 → -> instead of →
• Manual: entire explain --help | grep summary-timeout → "no automatic deadline applies"

Closes #1198 (issue was already closed at #1204 time; this delivers the diagnostic improvement that #1204 deferred).

---

Note

Medium Risk
Touches explain --generate execution/timeout behavior and adds a new streaming Claude CLI path; failures could affect summary generation UX and error reporting across providers.

Overview
entire explain --generate now reports live, phase-based progress during summary generation (TTY in-place updates, non-TTY line output, ASCII in ACCESSIBLE mode) and uses an observable-state summaryAttempt to produce more actionable timeout messages.

Adds optional agent.StreamingTextGenerator support (capability + detector) and implements Claude Code streaming via claude --output-format stream-json with NDJSON parsing, progress callbacks (connecting/TTFT/generating/done), and fallback to non-streaming on older CLIs.

Removes the implicit 5-minute default deadline for --generate (timeouts are now opt-in via flag/setting), and enhances non-streaming provider diagnostics by wrapping CLI errors with captured stderr and stdout byte counts (TextGenerationError), plus a shared provider→binary lookup (SummaryCLIBinaryName).

^{Reviewed by Cursor Bugbot for commit 252e320. Configure here.}

[Copilot]

Pull request overview

This PR upgrades entire explain --generate to use Claude CLI streaming output for incremental progress reporting, and introduces a TTY-gated deadline strategy (no deadline on interactive TTY by default; idle-based safety deadline in non-TTY).

Changes:

• Add a streaming text-generation capability (StreamingTextGenerator) and implement it for the Claude Code agent using --output-format stream-json with an NDJSON stream parser.
• Thread streaming progress events through summarization and render them in explain via summaryProgressWriter (TTY in-place updates vs. line-per-event for non-TTY/accessible mode).
• Replace the prior fixed wall-clock timeout behavior with a resolver-driven deadline policy plus an idle watchdog for non-TTY default behavior.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
cmd/entire/cli/explain.go	Adds progress rendering, deadline resolution, idle watchdog, and transcript-size preflight message for explain --generate.
cmd/entire/cli/explain_test.go	Updates and adds tests for progress writer, idle watchdog, and deadline policy behavior.
cmd/entire/cli/summarize/summarize.go	Extends GenerateFromTranscript with an optional progress callback and threads it into Claude generation.
cmd/entire/cli/summarize/summarize_test.go	Updates call sites for new signature and adds tests for progress callback threading.
cmd/entire/cli/summarize/claude.go	Prefers streaming generation when available and forwards progress callback into the agent.
cmd/entire/cli/summarize/claude_test.go	Adds test coverage for streaming preference/fallback in ClaudeGenerator.
cmd/entire/cli/strategy/manual_commit_condensation.go	Updates summarization call to match new GenerateFromTranscript signature.
cmd/entire/cli/agent/agent.go	Introduces progress phase/types and the StreamingTextGenerator interface.
cmd/entire/cli/agent/capabilities.go	Adds capability detection helper AsStreamingTextGenerator and declared-cap field.
cmd/entire/cli/agent/capabilities_test.go	Adds tests validating AsStreamingTextGenerator behavior.
cmd/entire/cli/agent/claudecode/generate.go	Switches Claude Code generation to streaming NDJSON parsing and emits progress callbacks.
cmd/entire/cli/agent/claudecode/response.go	Adds NDJSON stream event types and streamClaudeResponse parser.
cmd/entire/cli/agent/claudecode/response_test.go	Replaces legacy response parsing tests with stream parser tests using fixtures.
cmd/entire/cli/agent/claudecode/claude_test.go	Adds streaming-generation tests and delegation test for GenerateText.
cmd/entire/cli/agent/claudecode/testdata/stream_success.jsonl	Adds real-world success fixture for stream-json NDJSON.
cmd/entire/cli/agent/claudecode/testdata/stream_error_404.jsonl	Adds error fixture (is_error:true, 404) for stream-json NDJSON.

[peyton-alt]

@BugBot review

[peyton-alt]

@BugBot review

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 252e320. Configure here.}