NVIDIA · kevalmorabia97 · Jun 24, 2026 · Jun 22, 2026 · coderabbitai · Jun 24, 2026
@@ -18,7 +18,7 @@ If `MODELOPT_WORKSPACE_ROOT` is set, read `skills/common/workspace-management.md
 
 ```text
 - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set)
-- [ ] Step 1: Check `nel` install + existing config
+- [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs)
 - [ ] Step 2: Build base config (5-question flow OR shortcut)
 - [ ] Step 3: Configure deployment (model path, params, cross-check)
 - [ ] Step 4: Fill remaining ??? values
@@ -36,6 +36,10 @@ If `MODELOPT_WORKSPACE_ROOT` is set, read `skills/common/workspace-management.md
 
 Run `nel --version`; if missing, instruct `pip install nemo-evaluator-launcher`. If user has an existing config, skip to Step 8 (optionally review for `???` and quantization flags first).
 
+**Set up `.env` now (not Step 8).** The working `.env` lives at the **workspace root** — the directory you run `nel` from — matching `modelopttools:eval-config`'s convention; do **not** create it under the skill dir. (NEL does not discover `.env` by path: it reads secrets from the shell env via the `host:` prefix after you `source`, so the location is purely *which file you source* before `nel run`. Keeping the single `.env` at the workspace root avoids a stale duplicate under the symlinked, shared `.agents/` skill tree.) For judge-scored / user-sim tasks (HLE, AA-LCR, Tau2), seed it from the template if absent — the template ships under the skill dir, the working `.env` does not: `[ -f .env ] || cp .agents/skills/evaluation/recipes/env.example .env`. Then try `modelopttools:eval-config` (if available) to fill the judge `model_id`/`url` rows (user adds the secret key). Needed before Step 5, which substitutes those values into task `<VAR>` placeholders.
+
+**Secret safety — never open `.env` with Read/Write/Edit.** The harness mirrors later edits of any agent-opened file into the transcript, so touching `.env` leaks the keys the user adds afterward. Use shell only (`cp` to create, `source` to load — neither echoes); edit `env.example`, never `.env`; leave value entry to the user / `modelopttools:eval-config`.
+
 **Task recipes** (always read before editing the relevant task in the config):
 
 - AA Index v2 suite (default for quantized-checkpoint validation, see `references/quantization-benchmarks.md`): `recipes/tasks/aa/{gpqa_diamond,hle,lcr,scicode,ifbench,mmmu_pro,tau2_bench_telecom}.md`
@@ -95,7 +99,7 @@ Some models need extra vLLM backend env vars (model-card research) — e.g. `VLL
 
 #### Cross-check both sources for vLLM (mandatory, neither replaces the other)
 
-**Source 1 — `recipes.vllm.ai/<org>/<model>`** (curated vLLM recipes; authoritative for parallelism, family-specific flags like `--reasoning-parser` / `--tool-call-parser` / `--mm-encoder-tp-mode`, vLLM version, spec-decoding, GPU count). Pin variants via query params (e.g. `?variant=fp8&strategy=single_node_tep`).
+**Source 1 — `recipes.vllm.ai/<org>/<model>`** (curated vLLM recipes; authoritative for parallelism, family-specific flags like `--reasoning-parser` / `--tool-call-parser` / `--mm-encoder-tp-mode`, vLLM version, spec-decoding, GPU count). **Fetch the page for the EXACT model id, not a base/sibling** — variant minimums differ (e.g. MiniMax-M2 ≥0.11.0 vs M2.7 ≥0.20.0). Pin variants via query params (e.g. `?variant=fp8&strategy=single_node_tep`).
 
 > **WebFetch caveat — triage the summary:**
 >
@@ -142,14 +146,15 @@ Conventions: always start `vllm serve /checkpoint` (NEL mounts here); always `--
 
 For how to choose `--tensor-parallel-size` / `--data-parallel-size` / `--pipeline-parallel-size` (and EP) from the model size and your GPU count, read `references/parallelism.md` — cross-check the layout against `recipes.vllm.ai`, then adapt to the GPUs you actually have via the fit math there.
 
-**Image / vLLM version.** Default `image: vllm/vllm-openai:v0.19.1` (pinned for reproducibility). If `recipes.vllm.ai` states a higher minimum version for the chosen variant (e.g. "vLLM >= 0.20.0"), bump the image tag accordingly (e.g. `v0.20.0`) — do **not** stay on `0.19.1` when the recipe explicitly requires newer. Do **not** use `:latest` (drifts across re-runs, breaks reproducibility). The version is part of the cross-check: surface to the user when bumping.
+**Image / vLLM version.** Treat default `image: vllm/vllm-openai:v0.19.1` as a floor to verify: bump to the **exact model's** `recipes.vllm.ai` minimum if higher (e.g. `v0.20.0`). Running below minimum is a trap — the server starts, then a worker dies mid-inference with `CUDA error: an illegal memory access` (MiniMax-M2.7 NVFP4 needed ≥0.20.0), easy to misread as a kernel bug. Never `:latest` (breaks reproducibility). Surface version bumps to the user.
 
 > **NVFP4 on Blackwell B300/GB300 (sm_103): append `-cu130` to the image tag** (e.g. `vllm/vllm-openai:v0.19.1-cu130` — release tags are multi-arch). The default cu12 build has no sm_103 FP4 kernel, so engine init dies with `CUDA error: no kernel image is available`. If a pinned release predates the model's arch, use `cu130-nightly-<arch>` (Qwen3.5-9B's `qwen3_5` needed it, vLLM 0.19.2rc1.dev134). Multimodal on sm_103 may also need `--mm-encoder-attn-backend TRITON_ATTN`. Full note in `recipes/examples/example_eval.yaml`.
 
 #### vLLM-backend defaults — always include unless the recipe *contradicts*
 
 Silence is not contradiction. Drop/override only when the recipe sets a different value for the same setting (e.g. recipe pins `--max-num-batched-tokens 16384` → use 16384).
 
+- `--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 128}'` — **parallelizes checkpoint load**, the single biggest deploy-time cost for large checkpoints. A big MoE otherwise loads shards ~sequentially (~1 min/shard → e.g. ~40 min for a ~450 GB / 45-shard checkpoint); on a **preemptible** queue that long load window is exactly where jobs get killed before they ever serve. `num_threads` defaults to **128**; scale it to the checkpoint (smaller for small models, bounded by the shared-FS read bandwidth — too high yields no gain). Safe to always include.
 - `--max-num-batched-tokens 8192` — caps per-step batched tokens; prevents long-prefill stalls.
 - `--enable-chunked-prefill` — interleaves long prefills with decode steps (required for AA-LCR's ~120K input). Modern vLLM defaults this on for many models; set explicitly to avoid drift.
 - `--enable-expert-parallel` — **MoE-only default.** Detect MoE from handle suffix (`-A10B`, `-A3B`, etc.), `num_experts` / `num_local_experts` / `n_routed_experts` in `config.json`, or card. No-op when TP=DP=1, safe to always include for MoE. Do not add for dense models. See `references/parallelism.md` for what EP does and the DP-attention + EP-MoE throughput pattern.
@@ -261,7 +266,7 @@ Implications for the agent:
 
 4. Apply, show updated list, ask "Final, or more changes?" Loop until confirmed.
 
-**Tasks that call an external judge / user-simulator / scoring endpoint.** Treat this as a general pattern, not a fixed list — HLE, AA-LCR, and Tau2 need one today, but other benchmarks may too (check each task's recipe). Their `model_id` / `url` are **config, not secrets**: substitute the **literal** values the user keeps in `.env` (keys per the task's recipe + `recipes/env.example`) into the task's `<VAR>` placeholders. Do **not** emit `${oc.env:...}` for these (it silently fails unless the var was exported with `set -a`). Only `api_key` stays an env-var *name* (e.g. `INFERENCE_API_KEY`), exported and read by the harness. All nemo-skills/tau2 judges + user-sims (HLE, AA-LCR, Tau2) use one `INFERENCE_API_KEY` against one OpenAI-compatible host — *not* `build.nvidia.com`'s `JUDGE_API_KEY` (that's for simple-evals, e.g. AIME). Get the `*_MODEL_ID`/`*_URL` values from the `eval-config` skill if your org ships one, rather than guessing a host.
+**Tasks that call an external judge / user-simulator / scoring endpoint.** Treat this as a general pattern, not a fixed list — HLE, AA-LCR, and Tau2 need one today, but other benchmarks may too (check each task's recipe). Their `model_id` / `url` are **config, not secrets**: substitute the **literal** values the user keeps in `.env` (keys per the task's recipe + `recipes/env.example`) into the task's `<VAR>` placeholders. Do **not** emit `${oc.env:...}` for these (it silently fails unless the var was exported with `set -a`). Only `api_key` stays an env-var *name* (e.g. `INFERENCE_API_KEY`), exported and read by the harness. All nemo-skills/tau2 judges + user-sims (HLE, AA-LCR, Tau2) use one `INFERENCE_API_KEY` against one OpenAI-compatible host — *not* `build.nvidia.com`'s `JUDGE_API_KEY` (that's for simple-evals, e.g. AIME). Get the `*_MODEL_ID`/`*_URL` values via `modelopttools:eval-config` (see Step 1) rather than guessing a host; only fill them by hand if it's unavailable. `.env` should already exist (Step 1) — if not, set it up now (don't defer to Step 8) before substituting.
 
 **Known issue — nemo-skills self-deployment:** If using `nemo_skills.*` tasks (`ns_*`) with self-deployment (vLLM/SGLang/NIM), you need **both** of these:
 
@@ -319,10 +324,11 @@ Add credentials per `skills/common/slurm-setup.md` §6 if missing. If you can't
 
 Run directly when the user asked to launch; otherwise ask before submitting.
 
-**Env setup:** Copy `recipes/env.example` → `.env`, fill, source:
+**Env setup:** `.env` is normally already created and filled back in Step 1 (via `modelopttools:eval-config`), at the **workspace root** — the dir you run `nel` from, not under the skill dir. Ensure it exists and source it — do **not** clobber an existing `.env`:
 
 ```bash
-cp recipes/env.example .env
+# .env lives at the workspace root (where you run nel); the template ships under the skill dir
+[ -f .env ] || cp .agents/skills/evaluation/recipes/env.example .env   # create only if Step 1 didn't
 set -a && source .env && set +a
 
 # If pre_cmd/post_cmd in config (review pre_cmd first — runs arbitrary commands):
@@ -341,6 +347,8 @@ nel run --config <path> --dry-run
 
 Fix unresolved `???`, bad Hydra overrides, missing env vars, invalid mounts, image issues, sbatch errors, obvious deployment errors before proceeding.
 
+> **Dry-run does NOT validate the image/vLLM version** (image pulled only at deploy). Confirm `image:` ≥ the exact model's `recipes.vllm.ai` minimum (Step 3) before submitting — too-old passes dry-run, then crashes mid-inference.
+
 > **Non-fatal noise:** "Failed to get manifest"/`401`/`404`, "Could not extract frame definition file", "proceeding with minimal task definition", "Found N unlisted task(s)" — expected for `ns_*`/recipe tasks and private (gitlab) containers; the task still runs in-container. Set `NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1`. Real blockers: unresolved `???`, interpolation errors, bad mounts, sbatch rejections.
 
 **Step 8.2 — Canary** (limited-samples, validates everything dry-run can't):

@@ -1,7 +1,8 @@
 # Evaluation API Keys
 #
-# Copy this file and fill in the keys you need:
-#   cp recipes/env.example .env
+# Copy this file to your workspace root (the dir you run `nel` from) — NOT into
+# the skill dir — and fill in the keys you need:
+#   cp .agents/skills/evaluation/recipes/env.example .env
 #   # Edit .env with your keys
 #   set -a && source .env && set +a
 #
@@ -40,8 +41,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1
 # substituted as literal model_id/url into the config (matching <VAR> placeholders in
 # the recipes) — they do NOT need to be exported; only INFERENCE_API_KEY is.
 # URL note: nemo-skills uses the /v1 base; tau2-bench needs the full /v1/chat/completions.
-# If your org ships an `eval-config` skill, it fills the model_id/url values
-# below — otherwise point them at your own OpenAI-compatible judge host.
+# The `modelopttools:eval-config` skill fills the model_id/url values below (and
+# installs the optional launcher package) — invoke it for judge-scored runs;
+# otherwise point them at your own OpenAI-compatible judge host.
 
 # HLE judge (ns_hle_aa) — recommended GPT-4o
 # HLE_JUDGE_MODEL_ID=<judge-model-id>

@@ -6,7 +6,7 @@
 # references define benchmark requirements and YAML fragments, this file
 # defines the deployment + operational conventions.
 #
-# Includes (default): gpqa_diamond_aa_v3 (simple-evals harness, n_samples=16).
+# Includes (default): ns_gpqa (nemo-skills harness, num_repeats=16, AA mcq-4choices).
 # Add more AA tasks per recipes/tasks/aa/ and the AA suite rule in SKILL.md.
 #
 # Usage:
@@ -35,7 +35,7 @@
 # `huggingface_hub.snapshot_download`) and point `checkpoint_path` at it.
 #
 # Run a single task:
-#   nel run --config ... -t gpqa_diamond_aa_v3
+#   nel run --config ... -t ns_gpqa
 #
 # Canary (2 samples): use this before a full run to validate logs and tune
 # parallelism.
@@ -91,6 +91,9 @@ deployment:
   # model as `/checkpoint` and eval requests 404 ("model does not exist").
   # For MoE models, add `--enable-expert-parallel` to the command.
   # For models with custom code, add `--trust-remote-code` to the command.
+  # `--model-loader-extra-config` enables multithreaded checkpoint loading (on by
+  # default below) — cuts large-checkpoint load from tens of minutes to a few;
+  # tune num_threads to the checkpoint size / shared-FS read bandwidth.
   # After filling in evaluation `parallelism` values (top-level + per-task),
   # append `--max-num-seqs N` to the command where
   # N = ceil(max_parallelism / data_parallel_size).
@@ -102,6 +105,7 @@ deployment:
     --tensor-parallel-size 1
     --data-parallel-size 1
     --max-model-len 131072
+    --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 128}'
     --max-num-batched-tokens 8192
     --enable-chunked-prefill
 evaluation:
@@ -153,14 +157,17 @@ evaluation:
               # --max-model-len >= 393216. Uncomment for DeepSeek V4.
               # reasoning_effort: max
   tasks:
-  # Reasoning (chat endpoint, 8 repeats, short)
-    - name: gpqa_diamond_aa_v3
-      container: nvcr.io/nvidia/eval-factory/simple-evals:26.03
+  # Reasoning (chat endpoint, short). nemo-skills GPQA on the AA 4-choice MCQ
+  # prompt (golden Nemotron-3-Ultra harness); repeat count kept at this recipe's
+  # prior value of 16.
+    - name: ns_gpqa
+      container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
       nemo_evaluator_config:
         config:
           params:
             extra:
-              n_samples: 16
+              num_repeats: 16
+              args: "++prompt_config=eval/aai/mcq-4choices"
 
 export:
   # Use LITERAL values below — NOT ${deployment.*} / ${evaluation.*} cross-refs.

@@ -2,26 +2,31 @@
 
 ## Task Details
 
-- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/simple_evals.html#simple-evals-gpqa-diamond-aa-v3>
+- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/nemo_skills.html#nemo-skills-ns-gpqa>
 
 ## Params
 
+nemo-skills `ns_gpqa` on the AA 4-choice MCQ prompt
+(`++prompt_config=eval/aai/mcq-4choices`), `num_repeats: 16`.
+
 ## YAML Fragment
 
 Use this inside the top-level `evaluation.tasks` list:
 
 ```yaml
-- name: gpqa_diamond_aa_v3
-  container: nvcr.io/nvidia/eval-factory/simple-evals:26.03
+- name: ns_gpqa
+  container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
   nemo_evaluator_config:
     config:
       params:
         extra:
-          n_samples: 16
+          num_repeats: 16
+          args: "++prompt_config=eval/aai/mcq-4choices"
 ```
 
 ## Score Extraction from mlflow
 
-Result (0-100): `gpqa_diamond_score_micro_avg_of_N`
+Result (0-100): `gpqa_pass_at_1_avg-of-N_symbolic_correct`
 
-N is the repeat count.  If the repeat count is unknown, use the highest available `avg_of_N`.
+N is the repeat count (16 here, so `gpqa_pass_at_1_avg-of-16_symbolic_correct`).
+If the repeat count is unknown, use the highest available `avg-of-N`.
@@ -13,6 +13,14 @@ they're config, not secrets, so they don't need exporting. Only `api_key`
 (`INFERENCE_API_KEY`) is exported and read by the harness. Keep the judge fixed
 across comparable runs.
 
+`hle_strict_judge: true` (inside the `judge` block) enables strict judging.
+
+**Don't add `++server.enable_soft_fail=True` for a self-deployed model.** In
+nemo-skills 0.7.0 it forces the client to load a tokenizer from the served model
+name, which 404s when that name isn't a real HF repo (failing the run). If you
+need soft-fail, also set `++tokenizer` to an HF id or a container-loadable local
+tokenizer.
+
-`hle_strict_judge: true` (inside the `judge` block) enables strict judging.
-
-**Don't add `++server.enable_soft_fail=True` for a self-deployed model.** In
-nemo-skills 0.7.0 it forces the client to load a tokenizer from the served model
-name, which 404s when that name isn't a real HF repo (failing the run). If you
-need soft-fail, also set `++tokenizer` to an HF id or a container-loadable local
-tokenizer.
+`hle_strict_judge: true` (inside the `judge` block) enables strict judging.
+
+**For self-deployed models, only add `++server.enable_soft_fail=True` if you
+also set `++tokenizer` to an HF id or a container-loadable local tokenizer.**
+In nemo-skills 0.7.0, soft-fail otherwise makes the client load a tokenizer from
+the served model name, which can 404 if that name isn't a real HF repo.
-`hle_strict_judge: true` (inside the `judge` block) enables strict judging.
-
-**Don't add `++server.enable_soft_fail=True` for a self-deployed model.** In
-nemo-skills 0.7.0 it forces the client to load a tokenizer from the served model
-name, which 404s when that name isn't a real HF repo (failing the run). If you
-need soft-fail, also set `++tokenizer` to an HF id or a container-loadable local
-tokenizer.
+`hle_strict_judge: true` (inside the `judge` block) enables strict judging.
+
+**For self-deployed models, only add `++server.enable_soft_fail=True` if you
+also set `++tokenizer` to an HF id or a container-loadable local tokenizer.**
+In nemo-skills 0.7.0, soft-fail otherwise makes the client load a tokenizer from
+the served model name, which can 404 if that name isn't a real HF repo.
 ## YAML Fragment
 
 Use this inside the top-level `evaluation.tasks` list:
@@ -30,6 +38,7 @@ Use this inside the top-level `evaluation.tasks` list:
             model_id: <HLE_JUDGE_MODEL_ID>   # from .env; recommended GPT-4o
             url: <NS_JUDGE_URL>              # from .env (/v1 base)
             api_key: INFERENCE_API_KEY       # env-var name; exported, read by harness
+            hle_strict_judge: true
 ```
 
 ## Score Extraction from mlflow

@@ -2,19 +2,28 @@
 
 ## Task Details
 
-- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/simple_evals.html#simple-evals-mmlu-pro-aa-v3>
+- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/nemo_skills.html#nemo-skills-ns-mmlu-pro>
 
 ## Params
 
+nemo-skills `ns_mmlu_pro` on the AA 10-choice boxed MCQ prompt
+(`++prompt_config=eval/aai/mcq-10choices-boxed`), `num_repeats: 1`.
+
 ## YAML Fragment
 
 Use this inside the top-level `evaluation.tasks` list:
 
 ```yaml
-- name: mmlu_pro_aa_v3
-  container: nvcr.io/nvidia/eval-factory/simple-evals:26.03
+- name: ns_mmlu_pro
+  container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
+  nemo_evaluator_config:
+    config:
+      params:
+        extra:
+          num_repeats: 1
+          args: "++prompt_config=eval/aai/mcq-10choices-boxed"  # pragma: allowlist secret
 ```
 
-## Score Extraction
+## Score Extraction from mlflow
 
-Result (0-1): `mmlu_pro_score_micro`
+Result (0-100): `mmlu-pro_pass_at_1_symbolic_correct`  (note the hyphen in `mmlu-pro`)
@@ -137,7 +137,7 @@ default for short model-bound tasks and override the outliers:
 
 | Bottleneck | AA tasks | Cap by |
 | --- | --- | --- |
-| Model / GPU KV (short) | `gpqa_diamond_aa_v3`, `ns_ifbench` | top-level default (preemption-free KV-fit) |
+| Model / GPU KV (short) | `ns_gpqa`, `ns_ifbench` | top-level default (preemption-free KV-fit) |
 | Long-context KV (~120K) | `ns_aa_lcr` | **low** override — prefill thrash; MLA ≫ GQA |
 | Judge / user-sim rate limit | `ns_hle_aa`, `ns_aa_lcr`, `tau2_bench_telecom` | judge endpoint 429s, **not** the model |
 | Sandbox execution | `ns_scicode` | sandbox slots |
@@ -151,8 +151,8 @@ default for short model-bound tasks and override the outliers:
 
 ## Worked examples (8×B200)
 
-- **Dense 9B NVFP4** (~5–6 GB) → **TP=1/DP=8, no EP**. GPQA `n_samples=1` = 198 reqs
-  (request-bound) → `parallelism=256`, `max-num-seqs=32`. `n_samples=8` = 1584
+- **Dense 9B NVFP4** (~5–6 GB) → **TP=1/DP=8, no EP**. GPQA `num_repeats=1` = 198 reqs
+  (request-bound) → `parallelism=256`, `max-num-seqs=32`. `num_repeats=8` = 1584
   (capacity-bound) → start 512; tune up only while preemption≈0 (~82K reasoning output
   → knee may be <1024).
 - **Dense ~70B BF16, 8×H100/80GB** (~140 GB) → won't fit 1 GPU → **TP=2/DP=4, no EP**.