diff --git a/.agents/skills/evaluation/SKILL.md b/.agents/skills/evaluation/SKILL.md index 8f40a2d66c9..fac812d5a66 100644 --- a/.agents/skills/evaluation/SKILL.md +++ b/.agents/skills/evaluation/SKILL.md @@ -18,7 +18,7 @@ If `MODELOPT_WORKSPACE_ROOT` is set, read `skills/common/workspace-management.md ```text - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set) -- [ ] Step 1: Check `nel` install + existing config +- [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs) - [ ] Step 2: Build base config (5-question flow OR shortcut) - [ ] Step 3: Configure deployment (model path, params, cross-check) - [ ] Step 4: Fill remaining ??? values @@ -36,6 +36,10 @@ If `MODELOPT_WORKSPACE_ROOT` is set, read `skills/common/workspace-management.md Run `nel --version`; if missing, instruct `pip install nemo-evaluator-launcher`. If user has an existing config, skip to Step 8 (optionally review for `???` and quantization flags first). +**Set up `.env` now (not Step 8).** The working `.env` lives at the **workspace root** — the directory you run `nel` from — matching `modelopttools:eval-config`'s convention; do **not** create it under the skill dir. (NEL does not discover `.env` by path: it reads secrets from the shell env via the `host:` prefix after you `source`, so the location is purely *which file you source* before `nel run`. Keeping the single `.env` at the workspace root avoids a stale duplicate under the symlinked, shared `.agents/` skill tree.) For judge-scored / user-sim tasks (HLE, AA-LCR, Tau2), seed it from the template if absent — the template ships under the skill dir, the working `.env` does not: `[ -f .env ] || cp .agents/skills/evaluation/recipes/env.example .env`. Then try `modelopttools:eval-config` (if available) to fill the judge `model_id`/`url` rows (user adds the secret key). Needed before Step 5, which substitutes those values into task `` placeholders. + +**Secret safety — never open `.env` with Read/Write/Edit.** The harness mirrors later edits of any agent-opened file into the transcript, so touching `.env` leaks the keys the user adds afterward. Use shell only (`cp` to create, `source` to load — neither echoes); edit `env.example`, never `.env`; leave value entry to the user / `modelopttools:eval-config`. + **Task recipes** (always read before editing the relevant task in the config): - AA Index v2 suite (default for quantized-checkpoint validation, see `references/quantization-benchmarks.md`): `recipes/tasks/aa/{gpqa_diamond,hle,lcr,scicode,ifbench,mmmu_pro,tau2_bench_telecom}.md` @@ -95,7 +99,7 @@ Some models need extra vLLM backend env vars (model-card research) — e.g. `VLL #### Cross-check both sources for vLLM (mandatory, neither replaces the other) -**Source 1 — `recipes.vllm.ai//`** (curated vLLM recipes; authoritative for parallelism, family-specific flags like `--reasoning-parser` / `--tool-call-parser` / `--mm-encoder-tp-mode`, vLLM version, spec-decoding, GPU count). Pin variants via query params (e.g. `?variant=fp8&strategy=single_node_tep`). +**Source 1 — `recipes.vllm.ai//`** (curated vLLM recipes; authoritative for parallelism, family-specific flags like `--reasoning-parser` / `--tool-call-parser` / `--mm-encoder-tp-mode`, vLLM version, spec-decoding, GPU count). **Fetch the page for the EXACT model id, not a base/sibling** — variant minimums differ (e.g. MiniMax-M2 ≥0.11.0 vs M2.7 ≥0.20.0). Pin variants via query params (e.g. `?variant=fp8&strategy=single_node_tep`). > **WebFetch caveat — triage the summary:** > @@ -142,7 +146,7 @@ Conventions: always start `vllm serve /checkpoint` (NEL mounts here); always `-- For how to choose `--tensor-parallel-size` / `--data-parallel-size` / `--pipeline-parallel-size` (and EP) from the model size and your GPU count, read `references/parallelism.md` — cross-check the layout against `recipes.vllm.ai`, then adapt to the GPUs you actually have via the fit math there. -**Image / vLLM version.** Default `image: vllm/vllm-openai:v0.19.1` (pinned for reproducibility). If `recipes.vllm.ai` states a higher minimum version for the chosen variant (e.g. "vLLM >= 0.20.0"), bump the image tag accordingly (e.g. `v0.20.0`) — do **not** stay on `0.19.1` when the recipe explicitly requires newer. Do **not** use `:latest` (drifts across re-runs, breaks reproducibility). The version is part of the cross-check: surface to the user when bumping. +**Image / vLLM version.** Treat default `image: vllm/vllm-openai:v0.19.1` as a floor to verify: bump to the **exact model's** `recipes.vllm.ai` minimum if higher (e.g. `v0.20.0`). Running below minimum is a trap — the server starts, then a worker dies mid-inference with `CUDA error: an illegal memory access` (MiniMax-M2.7 NVFP4 needed ≥0.20.0), easy to misread as a kernel bug. Never `:latest` (breaks reproducibility). Surface version bumps to the user. > **NVFP4 on Blackwell B300/GB300 (sm_103): append `-cu130` to the image tag** (e.g. `vllm/vllm-openai:v0.19.1-cu130` — release tags are multi-arch). The default cu12 build has no sm_103 FP4 kernel, so engine init dies with `CUDA error: no kernel image is available`. If a pinned release predates the model's arch, use `cu130-nightly-` (Qwen3.5-9B's `qwen3_5` needed it, vLLM 0.19.2rc1.dev134). Multimodal on sm_103 may also need `--mm-encoder-attn-backend TRITON_ATTN`. Full note in `recipes/examples/example_eval.yaml`. @@ -150,6 +154,7 @@ For how to choose `--tensor-parallel-size` / `--data-parallel-size` / `--pipelin Silence is not contradiction. Drop/override only when the recipe sets a different value for the same setting (e.g. recipe pins `--max-num-batched-tokens 16384` → use 16384). +- `--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 128}'` — **parallelizes checkpoint load**, the single biggest deploy-time cost for large checkpoints. A big MoE otherwise loads shards ~sequentially (~1 min/shard → e.g. ~40 min for a ~450 GB / 45-shard checkpoint); on a **preemptible** queue that long load window is exactly where jobs get killed before they ever serve. `num_threads` defaults to **128**; scale it to the checkpoint (smaller for small models, bounded by the shared-FS read bandwidth — too high yields no gain). Safe to always include. - `--max-num-batched-tokens 8192` — caps per-step batched tokens; prevents long-prefill stalls. - `--enable-chunked-prefill` — interleaves long prefills with decode steps (required for AA-LCR's ~120K input). Modern vLLM defaults this on for many models; set explicitly to avoid drift. - `--enable-expert-parallel` — **MoE-only default.** Detect MoE from handle suffix (`-A10B`, `-A3B`, etc.), `num_experts` / `num_local_experts` / `n_routed_experts` in `config.json`, or card. No-op when TP=DP=1, safe to always include for MoE. Do not add for dense models. See `references/parallelism.md` for what EP does and the DP-attention + EP-MoE throughput pattern. @@ -261,7 +266,7 @@ Implications for the agent: 4. Apply, show updated list, ask "Final, or more changes?" Loop until confirmed. -**Tasks that call an external judge / user-simulator / scoring endpoint.** Treat this as a general pattern, not a fixed list — HLE, AA-LCR, and Tau2 need one today, but other benchmarks may too (check each task's recipe). Their `model_id` / `url` are **config, not secrets**: substitute the **literal** values the user keeps in `.env` (keys per the task's recipe + `recipes/env.example`) into the task's `` placeholders. Do **not** emit `${oc.env:...}` for these (it silently fails unless the var was exported with `set -a`). Only `api_key` stays an env-var *name* (e.g. `INFERENCE_API_KEY`), exported and read by the harness. All nemo-skills/tau2 judges + user-sims (HLE, AA-LCR, Tau2) use one `INFERENCE_API_KEY` against one OpenAI-compatible host — *not* `build.nvidia.com`'s `JUDGE_API_KEY` (that's for simple-evals, e.g. AIME). Get the `*_MODEL_ID`/`*_URL` values from the `eval-config` skill if your org ships one, rather than guessing a host. +**Tasks that call an external judge / user-simulator / scoring endpoint.** Treat this as a general pattern, not a fixed list — HLE, AA-LCR, and Tau2 need one today, but other benchmarks may too (check each task's recipe). Their `model_id` / `url` are **config, not secrets**: substitute the **literal** values the user keeps in `.env` (keys per the task's recipe + `recipes/env.example`) into the task's `` placeholders. Do **not** emit `${oc.env:...}` for these (it silently fails unless the var was exported with `set -a`). Only `api_key` stays an env-var *name* (e.g. `INFERENCE_API_KEY`), exported and read by the harness. All nemo-skills/tau2 judges + user-sims (HLE, AA-LCR, Tau2) use one `INFERENCE_API_KEY` against one OpenAI-compatible host — *not* `build.nvidia.com`'s `JUDGE_API_KEY` (that's for simple-evals, e.g. AIME). Get the `*_MODEL_ID`/`*_URL` values via `modelopttools:eval-config` (see Step 1) rather than guessing a host; only fill them by hand if it's unavailable. `.env` should already exist (Step 1) — if not, set it up now (don't defer to Step 8) before substituting. **Known issue — nemo-skills self-deployment:** If using `nemo_skills.*` tasks (`ns_*`) with self-deployment (vLLM/SGLang/NIM), you need **both** of these: @@ -319,10 +324,11 @@ Add credentials per `skills/common/slurm-setup.md` §6 if missing. If you can't Run directly when the user asked to launch; otherwise ask before submitting. -**Env setup:** Copy `recipes/env.example` → `.env`, fill, source: +**Env setup:** `.env` is normally already created and filled back in Step 1 (via `modelopttools:eval-config`), at the **workspace root** — the dir you run `nel` from, not under the skill dir. Ensure it exists and source it — do **not** clobber an existing `.env`: ```bash -cp recipes/env.example .env +# .env lives at the workspace root (where you run nel); the template ships under the skill dir +[ -f .env ] || cp .agents/skills/evaluation/recipes/env.example .env # create only if Step 1 didn't set -a && source .env && set +a # If pre_cmd/post_cmd in config (review pre_cmd first — runs arbitrary commands): @@ -341,6 +347,8 @@ nel run --config --dry-run Fix unresolved `???`, bad Hydra overrides, missing env vars, invalid mounts, image issues, sbatch errors, obvious deployment errors before proceeding. +> **Dry-run does NOT validate the image/vLLM version** (image pulled only at deploy). Confirm `image:` ≥ the exact model's `recipes.vllm.ai` minimum (Step 3) before submitting — too-old passes dry-run, then crashes mid-inference. + > **Non-fatal noise:** "Failed to get manifest"/`401`/`404`, "Could not extract frame definition file", "proceeding with minimal task definition", "Found N unlisted task(s)" — expected for `ns_*`/recipe tasks and private (gitlab) containers; the task still runs in-container. Set `NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1`. Real blockers: unresolved `???`, interpolation errors, bad mounts, sbatch rejections. **Step 8.2 — Canary** (limited-samples, validates everything dry-run can't): diff --git a/.agents/skills/evaluation/recipes/env.example b/.agents/skills/evaluation/recipes/env.example index 330d09c57ec..b1ed8310cdf 100644 --- a/.agents/skills/evaluation/recipes/env.example +++ b/.agents/skills/evaluation/recipes/env.example @@ -1,7 +1,8 @@ # Evaluation API Keys # -# Copy this file and fill in the keys you need: -# cp recipes/env.example .env +# Copy this file to your workspace root (the dir you run `nel` from) — NOT into +# the skill dir — and fill in the keys you need: +# cp .agents/skills/evaluation/recipes/env.example .env # # Edit .env with your keys # set -a && source .env && set +a # @@ -40,8 +41,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # substituted as literal model_id/url into the config (matching placeholders in # the recipes) — they do NOT need to be exported; only INFERENCE_API_KEY is. # URL note: nemo-skills uses the /v1 base; tau2-bench needs the full /v1/chat/completions. -# If your org ships an `eval-config` skill, it fills the model_id/url values -# below — otherwise point them at your own OpenAI-compatible judge host. +# The `modelopttools:eval-config` skill fills the model_id/url values below (and +# installs the optional launcher package) — invoke it for judge-scored runs; +# otherwise point them at your own OpenAI-compatible judge host. # HLE judge (ns_hle_aa) — recommended GPT-4o # HLE_JUDGE_MODEL_ID= diff --git a/.agents/skills/evaluation/recipes/examples/example_eval.yaml b/.agents/skills/evaluation/recipes/examples/example_eval.yaml index 6cc70811e9f..1810a2de3d5 100644 --- a/.agents/skills/evaluation/recipes/examples/example_eval.yaml +++ b/.agents/skills/evaluation/recipes/examples/example_eval.yaml @@ -6,7 +6,7 @@ # references define benchmark requirements and YAML fragments, this file # defines the deployment + operational conventions. # -# Includes (default): gpqa_diamond_aa_v3 (simple-evals harness, n_samples=16). +# Includes (default): ns_gpqa (nemo-skills harness, num_repeats=16, AA mcq-4choices). # Add more AA tasks per recipes/tasks/aa/ and the AA suite rule in SKILL.md. # # Usage: @@ -35,7 +35,7 @@ # `huggingface_hub.snapshot_download`) and point `checkpoint_path` at it. # # Run a single task: -# nel run --config ... -t gpqa_diamond_aa_v3 +# nel run --config ... -t ns_gpqa # # Canary (2 samples): use this before a full run to validate logs and tune # parallelism. @@ -91,6 +91,9 @@ deployment: # model as `/checkpoint` and eval requests 404 ("model does not exist"). # For MoE models, add `--enable-expert-parallel` to the command. # For models with custom code, add `--trust-remote-code` to the command. + # `--model-loader-extra-config` enables multithreaded checkpoint loading (on by + # default below) — cuts large-checkpoint load from tens of minutes to a few; + # tune num_threads to the checkpoint size / shared-FS read bandwidth. # After filling in evaluation `parallelism` values (top-level + per-task), # append `--max-num-seqs N` to the command where # N = ceil(max_parallelism / data_parallel_size). @@ -102,6 +105,7 @@ deployment: --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 131072 + --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 128}' --max-num-batched-tokens 8192 --enable-chunked-prefill evaluation: @@ -153,14 +157,17 @@ evaluation: # --max-model-len >= 393216. Uncomment for DeepSeek V4. # reasoning_effort: max tasks: - # Reasoning (chat endpoint, 8 repeats, short) - - name: gpqa_diamond_aa_v3 - container: nvcr.io/nvidia/eval-factory/simple-evals:26.03 + # Reasoning (chat endpoint, short). nemo-skills GPQA on the AA 4-choice MCQ + # prompt (golden Nemotron-3-Ultra harness); repeat count kept at this recipe's + # prior value of 16. + - name: ns_gpqa + container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03 nemo_evaluator_config: config: params: extra: - n_samples: 16 + num_repeats: 16 + args: "++prompt_config=eval/aai/mcq-4choices" export: # Use LITERAL values below — NOT ${deployment.*} / ${evaluation.*} cross-refs. diff --git a/.agents/skills/evaluation/recipes/tasks/aa/gpqa_diamond.md b/.agents/skills/evaluation/recipes/tasks/aa/gpqa_diamond.md index 8e15207558f..651aaae6ac2 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa/gpqa_diamond.md +++ b/.agents/skills/evaluation/recipes/tasks/aa/gpqa_diamond.md @@ -2,26 +2,31 @@ ## Task Details -- Reference: +- Reference: ## Params +nemo-skills `ns_gpqa` on the AA 4-choice MCQ prompt +(`++prompt_config=eval/aai/mcq-4choices`), `num_repeats: 16`. + ## YAML Fragment Use this inside the top-level `evaluation.tasks` list: ```yaml -- name: gpqa_diamond_aa_v3 - container: nvcr.io/nvidia/eval-factory/simple-evals:26.03 +- name: ns_gpqa + container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03 nemo_evaluator_config: config: params: extra: - n_samples: 16 + num_repeats: 16 + args: "++prompt_config=eval/aai/mcq-4choices" ``` ## Score Extraction from mlflow -Result (0-100): `gpqa_diamond_score_micro_avg_of_N` +Result (0-100): `gpqa_pass_at_1_avg-of-N_symbolic_correct` -N is the repeat count. If the repeat count is unknown, use the highest available `avg_of_N`. +N is the repeat count (16 here, so `gpqa_pass_at_1_avg-of-16_symbolic_correct`). +If the repeat count is unknown, use the highest available `avg-of-N`. diff --git a/.agents/skills/evaluation/recipes/tasks/aa/hle.md b/.agents/skills/evaluation/recipes/tasks/aa/hle.md index 0c2216cd5a7..713a1fe1fbc 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa/hle.md +++ b/.agents/skills/evaluation/recipes/tasks/aa/hle.md @@ -13,6 +13,14 @@ they're config, not secrets, so they don't need exporting. Only `api_key` (`INFERENCE_API_KEY`) is exported and read by the harness. Keep the judge fixed across comparable runs. +`hle_strict_judge: true` (inside the `judge` block) enables strict judging. + +**Don't add `++server.enable_soft_fail=True` for a self-deployed model.** In +nemo-skills 0.7.0 it forces the client to load a tokenizer from the served model +name, which 404s when that name isn't a real HF repo (failing the run). If you +need soft-fail, also set `++tokenizer` to an HF id or a container-loadable local +tokenizer. + ## YAML Fragment Use this inside the top-level `evaluation.tasks` list: @@ -30,6 +38,7 @@ Use this inside the top-level `evaluation.tasks` list: model_id: # from .env; recommended GPT-4o url: # from .env (/v1 base) api_key: INFERENCE_API_KEY # env-var name; exported, read by harness + hle_strict_judge: true ``` ## Score Extraction from mlflow diff --git a/.agents/skills/evaluation/recipes/tasks/mmlu_pro.md b/.agents/skills/evaluation/recipes/tasks/mmlu_pro.md index f2e40696991..228e8805121 100644 --- a/.agents/skills/evaluation/recipes/tasks/mmlu_pro.md +++ b/.agents/skills/evaluation/recipes/tasks/mmlu_pro.md @@ -2,19 +2,28 @@ ## Task Details -- Reference: +- Reference: ## Params +nemo-skills `ns_mmlu_pro` on the AA 10-choice boxed MCQ prompt +(`++prompt_config=eval/aai/mcq-10choices-boxed`), `num_repeats: 1`. + ## YAML Fragment Use this inside the top-level `evaluation.tasks` list: ```yaml -- name: mmlu_pro_aa_v3 - container: nvcr.io/nvidia/eval-factory/simple-evals:26.03 +- name: ns_mmlu_pro + container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03 + nemo_evaluator_config: + config: + params: + extra: + num_repeats: 1 + args: "++prompt_config=eval/aai/mcq-10choices-boxed" # pragma: allowlist secret ``` -## Score Extraction +## Score Extraction from mlflow -Result (0-1): `mmlu_pro_score_micro` +Result (0-100): `mmlu-pro_pass_at_1_symbolic_correct` (note the hyphen in `mmlu-pro`) diff --git a/.agents/skills/evaluation/references/parallelism.md b/.agents/skills/evaluation/references/parallelism.md index 06c937dbbb0..993691868e8 100644 --- a/.agents/skills/evaluation/references/parallelism.md +++ b/.agents/skills/evaluation/references/parallelism.md @@ -137,7 +137,7 @@ default for short model-bound tasks and override the outliers: | Bottleneck | AA tasks | Cap by | | --- | --- | --- | -| Model / GPU KV (short) | `gpqa_diamond_aa_v3`, `ns_ifbench` | top-level default (preemption-free KV-fit) | +| Model / GPU KV (short) | `ns_gpqa`, `ns_ifbench` | top-level default (preemption-free KV-fit) | | Long-context KV (~120K) | `ns_aa_lcr` | **low** override — prefill thrash; MLA ≫ GQA | | Judge / user-sim rate limit | `ns_hle_aa`, `ns_aa_lcr`, `tau2_bench_telecom` | judge endpoint 429s, **not** the model | | Sandbox execution | `ns_scicode` | sandbox slots | @@ -151,8 +151,8 @@ default for short model-bound tasks and override the outliers: ## Worked examples (8×B200) -- **Dense 9B NVFP4** (~5–6 GB) → **TP=1/DP=8, no EP**. GPQA `n_samples=1` = 198 reqs - (request-bound) → `parallelism=256`, `max-num-seqs=32`. `n_samples=8` = 1584 +- **Dense 9B NVFP4** (~5–6 GB) → **TP=1/DP=8, no EP**. GPQA `num_repeats=1` = 198 reqs + (request-bound) → `parallelism=256`, `max-num-seqs=32`. `num_repeats=8` = 1584 (capacity-bound) → start 512; tune up only while preemption≈0 (~82K reasoning output → knee may be <1024). - **Dense ~70B BF16, 8×H100/80GB** (~140 GB) → won't fit 1 GPU → **TP=2/DP=4, no EP**. diff --git a/.agents/skills/evaluation/references/quantization-benchmarks.md b/.agents/skills/evaluation/references/quantization-benchmarks.md index 518829f5223..f462b53ebec 100644 --- a/.agents/skills/evaluation/references/quantization-benchmarks.md +++ b/.agents/skills/evaluation/references/quantization-benchmarks.md @@ -17,10 +17,10 @@ to precision loss. The Artificial Analysis (AA) Index v2 suite under | Recipe | Benchmark | What it measures | Quant sensitivity | |--------|-----------|------------------|-------------------| -| `tasks/mmlu_pro.md` | MMLU-Pro (`mmlu_pro_aa_v3`, simple-evals) | General knowledge (10-choice) | Low — knowledge recall is robust to precision loss; cheap sanity check, not a regression detector | +| `tasks/mmlu_pro.md` | MMLU-Pro (`ns_mmlu_pro`, nemo-skills, `num_repeats: 1`) | General knowledge (10-choice boxed) | Low — knowledge recall is robust to precision loss; cheap sanity check, not a regression detector | | `tasks/aime_2025.md` | AIME 2025 (`AIME_2025_aa_v2`, simple-evals) | Competition math (`n_samples: 64`) | High — single-token errors in long chains-of-thought cascade into wrong final answers | | `tasks/livecodebench.md` | LiveCodeBench v6 (`ns_livecodebench`, nemo-skills) | Code generation (`num_repeats: 8`) | High — code is brittle to single-token errors (one wrong identifier = test failure) | -| `tasks/aa/gpqa_diamond.md` | GPQA Diamond (`gpqa_diamond_aa_v3`, simple-evals) | Hard science MCQ (`n_samples: 16`) | High — MCQ format but answers require multi-step reasoning that quantization can derail | +| `tasks/aa/gpqa_diamond.md` | GPQA Diamond (`ns_gpqa`, nemo-skills, `num_repeats: 16`) | Hard science MCQ (4-choice) | High — MCQ format but answers require multi-step reasoning that quantization can derail | | `tasks/aa/hle.md` | HLE | Humanity's Last Exam, text-only, judge-scored | High — hard reasoning at the frontier; small precision losses move borderline answers | | `tasks/aa/lcr.md` | LCR | Long-context reasoning (~120K input, judge-scored) | Very high — KV-cache and attention quant error accumulate across the full context window | | `tasks/aa/scicode.md` | SciCode | Multi-step scientific code + sandbox execution | Very high — reasoning + code + sandbox stacked; errors compound across subtasks | @@ -52,8 +52,8 @@ to precision loss. The Artificial Analysis (AA) Index v2 suite under - **Repeat / sample counts** in the task recipes are tuned for low variance — do **not** lower them for quant comparisons, or noise will mask real regressions. The field name differs by harness: `n_samples` for simple-evals - (AIME `64`, GPQA `16`) and tau2-bench (Tau2 `8`); `num_repeats` for - nemo-skills (AA-LCR `16`, LiveCodeBench/SciCode `8`, IFBench `5`). + (AIME `64`) and tau2-bench (Tau2 `8`); `num_repeats` for nemo-skills + (AA-LCR/GPQA `16`, LiveCodeBench/SciCode `8`, IFBench `5`, MMLU-Pro `1`). - **Judge / user-simulator endpoints** are required by AA-LCR, HLE AA, and Tau2-Bench Telecom. Keep the judge and (for Tau2) user-simulator models fixed across baseline and quantized runs for apples-to-apples comparison.