Linj router exp

.agents/skills/switchyard-lib-core/SKILL.md

@@ -39,7 +39,7 @@ right validation set. If the change is driven by a launcher need, also read
 | An OpenAI-compatible provider target such as NVIDIA Inference Hub or OpenRouter | Use the existing OpenAI-compatible backend/profile with `base_url`, `api_key`, and model id wiring. Add a new backend only when the provider has a real wire-format, auth, retry, or health contract that cannot fit that path. |
 | Direct Rust component bindings | Add concrete PyO3 classes under `crates/switchyard-py/src/component_bindings/`, keep config bindings near the component binding that consumes them, and expose them lazily from `switchyard_rust/components.py`. Do not keep growing `core_bindings.rs` or `switchyard_rust/core.py` with concrete component classes. |
 | Route YAML / model dispatch | Use `switchyard/cli/route_bundle.py` and `switchyard/lib/route_table_builders.py`. They build `RouteTable` entries from profile-backed runtimes and keep launchers plus `switchyard serve --routing-profiles` on one path. |
-| Shared/persistent session-affinity pins across workers or pod churn | Configure the latency route with `session_affinity: true` + `affinity_store: redis` + `affinity_store_url` (optional `affinity_store_ttl_seconds`, `affinity_key_prefix`). `SessionAffinity` keeps the Rust `SessionCache` as L1 and reads/writes through the `AffinityPinStore` L2 (`switchyard/lib/redis_pin_store.py`), fail-open behind a 0.1s socket timeout and a 3-failure/10s-cooldown circuit breaker (`switchyard_affinity_l2_breaker_open` gauge). Requires the `switchyard[affinity-redis]` extra. |
+| Shared/persistent session-affinity pins across workers or pod churn | Configure the latency route with `session_affinity: true` + `affinity_store: redis` + `affinity_store_url` (optional `affinity_store_ttl_seconds`, `affinity_key_prefix`); the escalation_router route takes the same `affinity_store*` keys (no `session_affinity` flag — its latch is always on; default prefix `swyd:esc:`). `SessionAffinity` keeps the Rust `SessionCache` as L1 and reads/writes through the `AffinityPinStore` L2 (`switchyard/lib/redis_pin_store.py`), fail-open behind a 0.1s socket timeout and a 3-failure/10s-cooldown circuit breaker (`switchyard_affinity_l2_breaker_open` gauge). Requires the `switchyard[affinity-redis]` extra. |
 | Stats / telemetry | Reuse `StatsRequestProcessor`, `StatsResponseProcessor`, `StatsLlmBackend`, and `StatsAccumulator`. A profile config should thread one accumulator through all three when stats are enabled. Do not write a parallel collector. |
 | A fixed-path endpoint contributed by per-route components | Set `Endpoint.register_once = True`; `build_switchyard_app(...)` mounts the first instance while still running every component's lifecycle. Leave the default `False` for configurable endpoint classes that may mount distinct instances. |
 | Per-endpoint attribution on `/metrics` for a Python backend that can't be wrapped by `StatsLlmBackend` | Set `ctx.selected_model = endpoint_id` before returning the response. Also set `ctx.backend_call_latency_ms = upstream_call_ms` so the response processor can compute routing overhead. `LatencyServiceLLMBackend.call` is the reference. |
@@ -62,9 +62,9 @@ Profile-owned construction keeps the full behavior in one reviewable module:
 5. Use `ProfileSwitchyard(config.build())` only when serving through the
    existing Python endpoint contract.
 
-Random routing, deterministic routing, plan-execute, passthrough, no-op,
-stage-router, latency-service, RouteLLM, and OSS-router profiles are the reference
-set under `switchyard/lib/profiles/`.
+Random routing, deterministic routing, escalation-router, plan-execute,
+passthrough, no-op, stage-router, latency-service, RouteLLM, and OSS-router
+profiles are the reference set under `switchyard/lib/profiles/`.
 
 ## Anti-Patterns
 
@@ -76,6 +76,7 @@ set under `switchyard/lib/profiles/`.
 | A new role-shaped abstraction invented outside the backend role. | Keep pre-call logic in request-side profile code or plain request components, call logic in `LLMBackend`, post-call logic in response-side profile code or plain response components, and final wire conversion in `TranslationEngine`. |
 | Adding an OpenRouter-specific backend or translator just to point at `https://openrouter.ai/api/v1`. | Route it through the OpenAI-compatible backend/profile and provider configuration. Keep provider-specific code for actual protocol differences. |
 | Making route YAML declare arbitrary Python processors. | Route YAML is deployment config for supported profile route types. Runtime-only hooks such as intake are injected by the caller through the route-table builder kwargs. |
+| Hand-billing token costs from raw stats fields — charging `prompt_tokens` at the fresh-input rate and adding `cached_tokens` on top. | `prompt_tokens` in stats snapshots is **total** input: `uncached + cached_tokens + cache_creation_tokens` (Anthropic's three sibling usage counters are normalized into it). Fresh input = `prompt − cached − creation`. Use the built-in estimators — the snapshot's own `cost_estimate`, `estimate_model_cost()` in `cost_estimator.py`, or Rust `stats/cost.rs` — instead of ad-hoc math; ignoring the subtraction overstates cost ~5× on cache-heavy agentic traffic. |
 
 ## When Something Genuinely Doesn't Fit
 

benchmark/compare_runs.py

@@ -0,0 +1,126 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Compare accuracy and token cost across run-baseline.sh run directories.
+
+Usage::
+
+    uv run --no-sync python benchmark/compare_runs.py \
+        label1=benchmark/tb_runs/<run-dir> label2=benchmark/tb_runs/<run-dir> ...
+
+Reads each run's Harbor ``result.json`` (accuracy) and Switchyard
+``routing_stats_final.json`` (per-tier tokens, judge/classifier overhead,
+cost estimate) and prints a side-by-side table plus per-task outcomes.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+
+def _load_run(run_dir: Path) -> dict[str, Any]:
+    results = sorted(run_dir.glob("jobs/*/result.json"))
+    if not results:
+        raise FileNotFoundError(f"no jobs/*/result.json under {run_dir}")
+    harbor = json.loads(results[0].read_text())
+    stats_path = run_dir / "routing_stats_final.json"
+    routing = json.loads(stats_path.read_text()) if stats_path.exists() else {}
+
+    ev = next(iter(harbor["stats"]["evals"].values()))
+    rewards: dict[str, float] = {}
+    for value, trials in (ev.get("reward_stats", {}).get("reward", {}) or {}).items():
+        for trial in trials:
+            # trial names look like <task>-<suffix>; strip the trial suffix.
+            rewards[trial.rsplit("-", 1)[0]] = float(value)
+
+    return {
+        "harbor": harbor["stats"],
+        "eval": ev,
+        "rewards": rewards,
+        "routing": routing,
+    }
+
+
+def _fmt_cost(routing: dict[str, Any]) -> tuple[float, float, float]:
+    ce = routing.get("cost_estimate", {}) or {}
+    return (
+        ce.get("total_cost", 0.0),
+        ce.get("backend_cost", 0.0),
+        ce.get("classifier_cost", 0.0),
+    )
+
+
+def main(argv: list[str]) -> int:
+    runs: dict[str, dict[str, Any]] = {}
+    for arg in argv:
+        label, _, path = arg.partition("=")
+        if not path:
+            print(f"ERROR: expected label=path, got {arg!r}")
+            return 2
+        runs[label] = _load_run(Path(path))
+
+    header = f"{'metric':<28}" + "".join(f"{label:>18}" for label in runs)
+    print(header)
+    print("-" * len(header))
+
+    def row(name: str, values: list[str]) -> None:
+        print(f"{name:<28}" + "".join(f"{v:>18}" for v in values))
+
+    row("completed trials", [str(r["harbor"]["n_completed_trials"]) for r in runs.values()])
+    row("errored trials", [str(r["harbor"]["n_errored_trials"]) for r in runs.values()])
+    row("mean reward", [f"{r['eval']['metrics'][0].get('mean', 0.0):.3f}" for r in runs.values()])
+    row(
+        "solved / total",
+        [
+            f"{sum(1 for v in r['rewards'].values() if v >= 1.0)}/{len(r['rewards'])}"
+            for r in runs.values()
+        ],
+    )
+    row(
+        "input tokens",
+        [f"{r['harbor'].get('n_input_tokens', 0):,}" for r in runs.values()],
+    )
+    row(
+        "output tokens",
+        [f"{r['harbor'].get('n_output_tokens', 0):,}" for r in runs.values()],
+    )
+    for label_idx, name in ((0, "total cost $"), (1, "backend cost $"), (2, "judge/clf cost $")):
+        row(name, [f"{_fmt_cost(r['routing'])[label_idx]:.3f}" for r in runs.values()])
+
+    # Tier split (escalation / routed runs only).
+    row(
+        "tier split (calls)",
+        [
+            " ".join(
+                f"{v.get('tier') or m.rsplit('/', 1)[-1]}:{v.get('calls', 0)}"
+                for m, v in (r["routing"].get("models", {}) or {}).items()
+            )
+            or "-"
+            for r in runs.values()
+        ],
+    )
+    clf_rows = []
+    for r in runs.values():
+        clf = r["routing"].get("classifier", {}) or {}
+        n, e = clf.get("total_requests", 0), clf.get("total_errors", 0)
+        clf_rows.append(f"{n} ({e} err)" if n or e else "-")
+    row("judge calls", clf_rows)
+
+    # Per-task outcome grid.
+    tasks = sorted({t for r in runs.values() for t in r["rewards"]})
+    print()
+    print(f"{'task':<44}" + "".join(f"{label:>14}" for label in runs))
+    for task in tasks:
+        marks = []
+        for r in runs.values():
+            v = r["rewards"].get(task)
+            marks.append("-" if v is None else ("PASS" if v >= 1.0 else "fail"))
+        print(f"{task:<44}" + "".join(f"{m:>14}" for m in marks))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

benchmark/routing-profiles/tb-lite-escalation-opus-deepseek-gemini.yaml

@@ -0,0 +1,35 @@
+routes:
+  switchyard:
+    type: escalation_router
+    enable_stats: true
+    fallback_target_on_evict: strong
+    # Distinguish k>1 repeated-task trials against one shared server: the
+    # session key includes the first 2 post-task messages, so trials diverge
+    # via early model responses instead of sharing an escalation latch.
+    # Requires nonzero sampling temperature; keep 0 outside benchmarks.
+    session_key_depth: 2
+    strong:
+      model: anthropic/claude-opus-4.7
+      api_key: ${OPENROUTER_API_KEY}
+      base_url: https://openrouter.ai/api/v1
+      format: openai
+      timeout_secs: 600
+      extra_body:
+        reasoning:
+          effort: high
+    weak:
+      model: deepseek/deepseek-v4-pro
+      api_key: ${OPENROUTER_API_KEY}
+      base_url: https://openrouter.ai/api/v1
+      format: openai
+      timeout_secs: 600
+    judge:
+      model: google/gemini-3.5-flash
+      api_key: ${OPENROUTER_API_KEY}
+      base_url: https://openrouter.ai/api/v1
+      timeout_secs: 5
+      min_turn: 3
+      recent_turn_window: 14
+      # Verdict dumps became opt-in upstream; the bench harness reads
+      # escalation_verdict= lines from the captured server log.
+      dump_verdicts: true

benchmark/routing-profiles/tb-lite-escalation-opus-deepseek-nvidia.yaml

@@ -0,0 +1,73 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Escalation-router Opus/DeepSeek profile for Harbor Terminal-Bench Lite on
+# the NVIDIA Inference Hub. Every conversation starts on DeepSeek V4 Pro; a
+# DeepSeek V4 Flash judge watches the trajectory and latches the session to
+# Claude Opus 4.8 on a clear pattern of trouble.
+#
+# All three tiers use https://inference-api.nvidia.com/v1, which requires a
+# LiteLLM virtual key (sk-...), not an nvapi- key. Export the same key for
+# all three vars (they are the ones run-baseline.sh forwards into the
+# Switchyard container):
+#   export SWITCHYARD_STRONG_API_KEY=sk-...
+#   export SWITCHYARD_WEAK_API_KEY=sk-...
+#   export SWITCHYARD_CLASSIFIER_API_KEY=sk-...
+#
+# Run with:
+#   benchmark/run-baseline.sh \
+#     --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
+#     --routing-profiles benchmark/routing-profiles/tb-lite-escalation-opus-deepseek-nvidia.yaml \
+#     --model switchyard
+
+routes:
+  switchyard:
+    type: escalation_router
+    enable_stats: true
+    fallback_target_on_evict: strong
+    # Distinguish k>1 repeated-task trials against one shared server: the
+    # session key includes the first 2 post-task messages, so trials diverge
+    # via early model responses instead of sharing an escalation latch.
+    # Requires nonzero sampling temperature; keep 0 outside benchmarks.
+    session_key_depth: 2
+    strong:
+      model: aws/anthropic/bedrock-claude-opus-4-8
+      api_key: ${SWITCHYARD_STRONG_API_KEY}
+      base_url: https://inference-api.nvidia.com/v1
+      format: openai
+      timeout_secs: 600
+    weak:
+      # The evals- batch-gateway variant (nvidia/deepseek-ai/evals-deepseek-v4-pro)
+      # avoids the regular gateway's 6-min timeout under high concurrency, but
+      # its upstream host is currently failing TLS verification; switch back
+      # once it probes clean.
+      model: nvidia/deepseek-ai/deepseek-v4-pro
+      api_key: ${SWITCHYARD_WEAK_API_KEY}
+      base_url: https://inference-api.nvidia.com/v1
+      format: openai
+      timeout_secs: 600
+      # Explicit extra_body overrides the DeepSeek default of
+      # enable_thinking: false; the X-Inference-Priority: batch header is
+      # still injected (headers merge independently of the body).
+      extra_body:
+        chat_template_kwargs:
+          enable_thinking: true
+    judge:
+      model: nvidia/deepseek-ai/deepseek-v4-flash
+      api_key: ${SWITCHYARD_CLASSIFIER_API_KEY}
+      base_url: https://inference-api.nvidia.com/v1
+      # Hub Flash latency is ~4-8s on small judge calls (and the profile
+      # build pins DeepSeek judges to the X-Inference-Priority: batch
+      # gateway); the 5s default used with Gemini Flash would fail-open
+      # (stay weak) on most turns here.
+      timeout_secs: 30
+      min_turn: 3
+      # Require a second consecutive escalate verdict before latching:
+      # round-2 traces showed one-shot eager verdicts (single failed
+      # command at turn 4-9) causing regressions on tasks the weak tier
+      # solves; real loops persist across turns and still latch.
+      confirmations: 2
+      recent_turn_window: 14
+      # Verdict dumps became opt-in upstream; the bench harness reads
+      # escalation_verdict= lines from the captured server log.
+      dump_verdicts: true

benchmark/routing-profiles/tb-lite-single-deepseek-v4-pro-nvidia.yaml

@@ -0,0 +1,34 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Serve-backed single-model DeepSeek V4 Pro profile on the NVIDIA Inference
+# Hub — weak-only baseline for the escalation-router comparison.
+# Requires a LiteLLM virtual key (sk-...):
+#   export SWITCHYARD_WEAK_API_KEY=sk-...
+# Run with:
+#   benchmark/run-baseline.sh \
+#     --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
+#     --routing-profiles benchmark/routing-profiles/tb-lite-single-deepseek-v4-pro-nvidia.yaml \
+#     --model tb-lite-single-deepseek-v4-pro-nvidia
+
+defaults:
+  api_key: ${SWITCHYARD_WEAK_API_KEY}
+  base_url: https://inference-api.nvidia.com/v1
+  timeout_secs: 600.0
+
+routes:
+  tb-lite-single-deepseek-v4-pro-nvidia:
+    type: model
+    target:
+      # The evals- batch-gateway variant avoids the regular gateway's 6-min
+      # timeout under high concurrency but is currently failing TLS
+      # verification upstream; switch back once it probes clean.
+      model: nvidia/deepseek-ai/deepseek-v4-pro
+      format: openai
+      extra_headers:
+        X-Inference-Priority: batch
+      # Thinking on, matching the escalation profile's weak tier so the
+      # weak-only baseline stays apples-to-apples.
+      extra_body:
+        chat_template_kwargs:
+          enable_thinking: true

benchmark/routing-profiles/tb-lite-single-opus-4-7-nvidia.yaml

@@ -0,0 +1,24 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Serve-backed single-model Opus 4.7 profile on the NVIDIA Inference Hub —
+# strong-only baseline for the escalation-router comparison.
+# Requires a LiteLLM virtual key (sk-...):
+#   export SWITCHYARD_STRONG_API_KEY=sk-...
+# Run with:
+#   benchmark/run-baseline.sh \
+#     --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
+#     --routing-profiles benchmark/routing-profiles/tb-lite-single-opus-4-7-nvidia.yaml \
+#     --model tb-lite-single-opus-4-7-nvidia
+
+defaults:
+  api_key: ${SWITCHYARD_STRONG_API_KEY}
+  base_url: https://inference-api.nvidia.com/v1
+  timeout_secs: 600.0
+
+routes:
+  tb-lite-single-opus-4-7-nvidia:
+    type: model
+    target:
+      model: aws/anthropic/bedrock-claude-opus-4-7
+      format: openai

benchmark/routing-profiles/tb-lite-single-opus-4-8-nvidia.yaml

@@ -0,0 +1,24 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Serve-backed single-model Opus 4.8 profile on the NVIDIA Inference Hub —
+# strong-only baseline for the escalation-router comparison.
+# Requires a LiteLLM virtual key (sk-...):
+#   export SWITCHYARD_STRONG_API_KEY=sk-...
+# Run with:
+#   benchmark/run-baseline.sh \
+#     --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
+#     --routing-profiles benchmark/routing-profiles/tb-lite-single-opus-4-8-nvidia.yaml \
+#     --model tb-lite-single-opus-4-8-nvidia
+
+defaults:
+  api_key: ${SWITCHYARD_STRONG_API_KEY}
+  base_url: https://inference-api.nvidia.com/v1
+  timeout_secs: 600.0
+
+routes:
+  tb-lite-single-opus-4-8-nvidia:
+    type: model
+    target:
+      model: aws/anthropic/bedrock-claude-opus-4-8
+      format: openai

crates/switchyard-components/src/stats/cost.rs

@@ -189,11 +189,28 @@ fn raw_model_price(model: &str) -> Option<ModelPrice> {
             cached: 0.15,
             cache_write: 1.50,
         },
-        "aws/anthropic/bedrock-claude-opus-4-7"
+        // Nemotron 3 Ultra (550B/55B MoE) — OpenRouter reference list
+        // price, June 2026. evals-N ids are parallel benchmarking-gateway
+        // deployments of the same model.
+        "nvidia/nvidia/nemotron-3-ultra"
+        | "openai/nvidia/nvidia/nemotron-3-ultra"
+        | "nvidia/nvidia/evals-nemotron-ultra"
+        | "nvidia/nvidia/evals-nemotron-ultra-2"
+        | "nvidia/nvidia/evals-nemotron-ultra-3"
+        | "nvidia/nvidia/evals-nemotron-ultra-4" => ModelPrice {
+            input: 0.50,
+            output: 2.20,
+            cached: 0.05,
+            cache_write: 0.50,
+        },
+        "aws/anthropic/bedrock-claude-opus-4-8"
+        | "aws/anthropic/bedrock-claude-opus-4-7"
         | "aws/anthropic/bedrock-claude-opus-4-6"
         | "aws/anthropic/bedrock-claude-opus-4-5"
+        | "azure/anthropic/claude-opus-4-8"
         | "azure/anthropic/claude-opus-4-7"
         | "azure/anthropic/claude-opus-4-6"
+        | "claude-opus-4-8"
         | "claude-opus-4-7"
         | "claude-opus-4-6"
         | "claude-opus-4-5" => ModelPrice {
@@ -204,6 +221,7 @@ fn raw_model_price(model: &str) -> Option<ModelPrice> {
         },
         "aws/anthropic/bedrock-claude-sonnet-4-6"
         | "aws/anthropic/bedrock-claude-sonnet-4-5"
+        | "azure/anthropic/claude-sonnet-4-5"
         | "claude-sonnet-4-6"
         | "claude-sonnet-4-5" => ModelPrice {
             input: 3.00,