feat(launch): startup-timing breakdown and faster AUTO backend-format detection

[elyasmnvidian]

What this adds

Tooling and a fix for slow switchyard launch claude startup, tied to SWITCH-872.

TL;DR — AUTO backend-format detection is the startup cost. With format: auto (the Claude Code / Codex launcher default), the resolver probes the upstream live before the agent starts to detect the wire format — Chat Completions first, then /v1/messages, then /v1/responses. Each probe is a round-trip, so a slow endpoint (e.g. a cold NVCF/Azure deployment on Inference Hub) adds real wall-clock latency to startup. This PR reduces the worst case and adds the tooling to see it.

1. An opt-in, per-probe startup-timing breakdown. Enable it two ways — the --startup-timing flag or the SWITCHYARD_STARTUP_TIMING=1 env var; if neither is set it does nothing and a normal launch pays zero. When on, the launcher prints to stderr, right before it spawns the agent, how long each startup phase took — each backend-format probe on its own line:

switchyard startup timing:
       2.0 ms  config + credentials resolved
       0.1 ms  chain init
    1174.0 ms  probe: /v1/chat/completions
       0.4 ms  chain assembled
      47.2 ms  proxy thread started
      69.5 ms  proxy health-ready
       0.3 ms  child agent spawned
  --------
    1293.5 ms  total (launch invoked -> child spawn)

A slow line names the slow phase — three slow probe: lines mean a slow upstream made AUTO stack timeouts. Works for both single-model (--model) and default classifier-routing launches.

2. A fix for the phase that breakdown exposed. Before this change, if the Chat Completions probe timed out (reachable-but-slow endpoint), the resolver treated the timeout like a 404 and serially ran the /v1/messages and /v1/responses probes too, each stacking another timeout — one slow endpoint became several seconds (measured up to ~7 s). Now a probe timeout is detected via the raised httpx.TimeoutException (not a wall-clock heuristic) and falls back to Chat Completions immediately. A genuine fast 404 still falls through to the other probes, so detection is unchanged for Anthropic-native / Responses-only endpoints. Worst case drops from ~three stacked timeouts to one.

3. Docs. docs/architecture.md now states that AUTO costs startup latency (a live probe per format) and that setting format: explicitly removes it — with a pointer to --startup-timing to measure it. Also corrected the AUTO decision-tree diagram, which showed /v1/messages probed first; the resolver actually probes /v1/chat/completions first (with the timeout short-circuit).

What stays the same

• Explicit format: values still skip probing. Setting format: is the real fix for slow startup.
• anthropic/ and claude… model prefixes still short-circuit to Anthropic with no probe.
• A fast 404 on Chat Completions still falls through to the /v1/messages and /v1/responses probes. Only a probe timeout changes behavior.

Tests

tests/test_backend_format_resolver.py:

• A Chat Completions probe that raises httpx.ReadTimeout resolves to OPENAI without running the other two probes.
• AUTO records each probe as its own startup-timing mark.

The existing probe-order tests are unchanged. The --startup-timing output is debug-only diagnostic tooling, so it ships without its own test.

Not included

• The one-time Python import (~1 s) runs before launch dispatch, so it is not in the breakdown. Measure it with switchyard launch claude --model X --dry-run. Trimming it is separate work.
• No format cache. A stale cached format would misroute silently, and the timeout fix already removes the slow-startup case a cache would paper over.

[coderabbitai]

Walkthrough

Changes

Launch startup timing

Layer / File(s)	Summary
Startup timing utility and coverage switchyard/cli/launchers/startup_timing.py, tests/test_startup_timing.py	Adds opt-in environment-controlled timing marks, elapsed-stage output, state reset behavior, and pytest coverage.
Launch timing checkpoints switchyard/cli/launch_command.py, switchyard/cli/launchers/claude_code_launcher.py	Records launch, configuration, chain, proxy, readiness, and child-process checkpoints before dumping timing output.

Backend probe timeout handling

Layer / File(s)	Summary
Chat Completions timeout fallback switchyard/lib/backends/backend_format_resolver.py, tests/test_backend_format_resolver.py	Measures the initial Chat Completions probe and assumes BackendFormat.OPENAI when its timeout budget is exceeded, with a regression test.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

I’m a rabbit with a stopwatch bright,
Marking launch steps through the night.
Probes that linger lose their way,
OpenAI takes the reins today.
Ticks are cleared when prints are done—
Hop, hop, startup race is won!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the two main changes: startup timing breakdown and faster AUTO backend-format detection.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

switchyard/lib/backends/backend_format_resolver.py (1)

89-114: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Use the probe result instead of elapsed time to detect a timeout. probe_openai_chat_completions_support_sync() collapses all httpx.RequestError subtypes into False, so time.perf_counter() - chat_started >= timeout_s is guessing. HTTPX applies connect/read/write/pool timeouts independently, so a slow but successful fast-negative can exceed timeout_s, and a real timeout can still miss the threshold. Return a distinct timeout signal from the probe and branch on that instead.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@switchyard/lib/backends/backend_format_resolver.py` around lines 89 - 114,
Update probe_openai_chat_completions_support_sync and the surrounding resolver
flow to return a distinct timeout result instead of collapsing timeout
RequestErrors into False. In the Chat Completions branch, remove the
chat_started elapsed-time heuristic and branch directly on that timeout signal,
while preserving successful probes and fast-negative fallthrough behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@switchyard/cli/launchers/claude_code_launcher.py`:
- Around line 393-394: Move the “child agent spawned” timing checkpoint from the
pre-invocation location into the actual child-process creation path used by
ShellTUI or _supervise_claude_plain, so the timestamp and dumped total include
process creation. Preserve the existing timing flow otherwise, and add a
regression test verifying the checkpoint occurs after the child is spawned.

---

Outside diff comments:
In `@switchyard/lib/backends/backend_format_resolver.py`:
- Around line 89-114: Update probe_openai_chat_completions_support_sync and the
surrounding resolver flow to return a distinct timeout result instead of
collapsing timeout RequestErrors into False. In the Chat Completions branch,
remove the chat_started elapsed-time heuristic and branch directly on that
timeout signal, while preserving successful probes and fast-negative fallthrough
behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c6519c72-3901-427e-9a56-86703ec4c271

📥 Commits

Reviewing files that changed from the base of the PR and between e62fe46 and d0f8700.

📒 Files selected for processing (6)

• switchyard/cli/launch_command.py
• switchyard/cli/launchers/claude_code_launcher.py
• switchyard/cli/launchers/startup_timing.py
• switchyard/lib/backends/backend_format_resolver.py
• tests/test_backend_format_resolver.py
• tests/test_startup_timing.py

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-70/
Built to branch gh-pages at 2026-07-17 18:20 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[elyasmnvidian]

Thanks @coderabbitai — addressing both findings:

backend_format_resolver.py (timeout via elapsed time, Major): already fixed. The stopwatch heuristic was reviewed on an earlier commit; a later commit (40d8a6d6) removed the time.perf_counter() elapsed-time check entirely. The probe now re-raises httpx.TimeoutException, and _resolve_auto branches on the caught exception — exactly the distinct-timeout-signal approach suggested. No import time remains in the resolver.

claude_code_launcher.py (move "child agent spawned" mark into process creation): intentionally left where it is. The mark and dump() sit just before the handoff to ShellTUI.run() / _supervise_claude_plain() on purpose: once the child grabs the terminal (interactive TUI), stderr output would interleave with the child's UI. The timing is meant to capture switchyard's own pre-spawn overhead, which ends exactly at this handoff — exec of the child is not switchyard-attributable. Adding a mark after process creation would print into the running child's terminal.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -e
git log --oneline -15 origin/main..HEAD 2>/dev/null || git log --oneline -15
echo "---"
git log --all --oneline | grep -i "40d8a6d6" || true
echo "---search-file---"
fd backend_format_resolver.py
echo "---content---"
cat -n switchyard/lib/backends/backend_format_resolver.py

Length of output: 19817

---

Confirmed both points against the current HEAD.

@elyasmnvidian Thanks for the follow-up — I verified both points against the current HEAD.

1. backend_format_resolver.py timeout handling: Confirmed. Commit 40d8a6d (fix(backends): detect AUTO probe timeout via exception, not stopwatch) removed the time.perf_counter() elapsed-time heuristic entirely. The current code has no import time, and probe_openai_chat_completions_support_sync re-raises httpx.TimeoutException on a transport timeout while _resolve_auto catches it explicitly to return BackendFormat.OPENAI with reason "chat-completions probe timed out; assuming Chat Completions". This is exactly the distinct-signal approach I suggested — no ambiguity between a slow-but-successful probe and a real timeout.

2. claude_code_launcher.py mark placement: That's a reasonable rationale — keeping the final mark and dump() strictly before the handoff to the interactive child (ShellTUI.run() / _supervise_claude_plain()) avoids interleaving timing output with the child's own TUI rendering, and it correctly scopes the measurement to switchyard's own pre-spawn overhead rather than the child's lifetime. I'm fine leaving it as-is.

(‑, ‑)ゞ