From adc36ebcfe1397e5eba125416c801f6fe51a27bb Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Fri, 26 Jun 2026 16:45:59 +0000 Subject: [PATCH 1/4] Add AA-Omniscience eval recipe; harden judge/run conventions - Add recipes/tasks/aa/omniscience.md (AA Index v2; gemini-3-flash-preview judge; num_repeats=10; score omniscience_pass_at_1_avg-of-N_judge_correct) and list it in the AA Index v2 suite in SKILL.md. - Hardcode judge model_ids in the HLE / AA-LCR / Tau2 recipes (with a swap-for-equivalent note) and rename the shared judge URL var NS_JUDGE_URL -> INFERENCE_JUDGE_URL; fold the judge key into INFERENCE_API_KEY in env.example. - example_eval.yaml: add the OccupiedIdleGPUsJobReaper sbatch_comment exemption so eval jobs are not reaped during low-GPU-utilization phases (model load, judge calls, aggregation). - SKILL.md: document manual resume after a preemption / idle-reaper kill by re-submitting the job's run.sub (skip_filled -> cumulative progress). Signed-off-by: Chenjie Luo --- .agents/skills/evaluation/SKILL.md | 10 ++++- .agents/skills/evaluation/recipes/env.example | 34 ++++++-------- .../recipes/examples/example_eval.yaml | 2 + .../skills/evaluation/recipes/tasks/aa/hle.md | 16 +++---- .../skills/evaluation/recipes/tasks/aa/lcr.md | 13 +++--- .../recipes/tasks/aa/omniscience.md | 44 +++++++++++++++++++ .../recipes/tasks/aa/tau2_bench_telecom.md | 17 +++---- 7 files changed, 93 insertions(+), 43 deletions(-) create mode 100644 .agents/skills/evaluation/recipes/tasks/aa/omniscience.md diff --git a/.agents/skills/evaluation/SKILL.md b/.agents/skills/evaluation/SKILL.md index fac812d5a66..5b36002ec8e 100644 --- a/.agents/skills/evaluation/SKILL.md +++ b/.agents/skills/evaluation/SKILL.md @@ -42,7 +42,7 @@ Run `nel --version`; if missing, instruct `pip install nemo-evaluator-launcher`. **Task recipes** (always read before editing the relevant task in the config): -- AA Index v2 suite (default for quantized-checkpoint validation, see `references/quantization-benchmarks.md`): `recipes/tasks/aa/{gpqa_diamond,hle,lcr,scicode,ifbench,mmmu_pro,tau2_bench_telecom}.md` +- AA Index v2 suite (default for quantized-checkpoint validation, see `references/quantization-benchmarks.md`): `recipes/tasks/aa/{gpqa_diamond,hle,lcr,scicode,ifbench,mmmu_pro,tau2_bench_telecom,omniscience}.md` - Optional: `recipes/tasks/mmlu_pro.md`, `recipes/tasks/aime_2025.md`, `recipes/tasks/livecodebench.md` **AA rule:** If the user mentions "AA" / "Artificial Analysis", generate **only** tasks under `recipes/tasks/aa/`. Do not add MMLU-Pro, AIME 2025, or LiveCodeBench unless explicitly asked. @@ -240,6 +240,14 @@ On SLURM, several deploy/eval failures are invisible to `--dry-run` and only sur Evals that exceed 4h of wall-clock time are handled by **NEL's built-in dependency-chain resume**, not by shrinking the eval. NEL submits the first SLURM job; if it hits walltime, a dependent follow-on job resumes from the response/result caches the first job wrote, then queues another follow-on. Long evals continue across walltime windows automatically. See `references/run-validation.md#nel-timeout-and-resume-behavior` for the full mechanism. +**Preemption / external kill — resume manually with `sbatch run.sub`.** On a preemptible account (common on busy internal clusters) the scheduler can **CANCEL** a run mid-eval for a higher-priority job — `sacct -j ` shows `CANCELLED by ` (a `svc-*` service account) with `Elapsed` well under the 4h walltime. NEL does **not** auto-resume this (its dependency chain only fires on a genuine walltime timeout). But the job's generated `run.sub` is **re-submittable** and resumes from the same `output_dir` + response cache (`skip_filled`), continuing from the partial output rather than restarting: + +```bash +ssh "cd /-// && sbatch run.sub" +``` + +Re-submit again if it's preempted again — each resume re-deploys, then skips already-generated samples, so progress is **cumulative** across attempts until it completes. Always confirm via `sacct -j ` that the prior job was `CANCELLED` (not a real failure) before resuming. + Implications for the agent: - Do **not** lower `num_repeats`, split heavy tasks (AA-LCR, SciCode) into separate configs, or otherwise carve up the eval to fit inside 4h. Let NEL chain. diff --git a/.agents/skills/evaluation/recipes/env.example b/.agents/skills/evaluation/recipes/env.example index b1ed8310cdf..edf6d38eccd 100644 --- a/.agents/skills/evaluation/recipes/env.example +++ b/.agents/skills/evaluation/recipes/env.example @@ -33,27 +33,21 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # JUDGE_API_KEY= # INFERENCE_API_KEY= -# --- Optional: judge / user-simulator endpoints (model_id + URL) --- +# --- Optional: judge / user-simulator endpoints (URL only) --- # -# External judge / user-simulator / scoring endpoints, for any task that needs one -# (HLE, AA-LCR, Tau2 below — add more for other such benchmarks; auth via -# INFERENCE_API_KEY above). These are config, not secrets: the values you set here are -# substituted as literal model_id/url into the config (matching placeholders in -# the recipes) — they do NOT need to be exported; only INFERENCE_API_KEY is. -# URL note: nemo-skills uses the /v1 base; tau2-bench needs the full /v1/chat/completions. -# The `modelopttools:eval-config` skill fills the model_id/url values below (and -# installs the optional launcher package) — invoke it for judge-scored runs; -# otherwise point them at your own OpenAI-compatible judge host. - -# HLE judge (ns_hle_aa) — recommended GPT-4o -# HLE_JUDGE_MODEL_ID= -# AA-LCR judge (ns_aa_lcr) — recommended Qwen3 235B -# LCR_JUDGE_MODEL_ID= -# NS_JUDGE_URL=https:///v1 # shared by both judges above - -# Tau2 (tau2_bench_telecom) — user-sim Qwen3 235B, judger gpt-oss-120B -# TAU2_USER_MODEL_ID= -# TAU2_JUDGER_MODEL_ID= +# Judge `model_id`s are now HARDCODED in the task recipes (HLE azure/openai/gpt-4o, +# AA-LCR nvidia/qwen/qwen-235b, Tau2 judger nvidia/openai/gpt-oss-120b, +# AA-Omniscience gcp/google/gemini-3-flash-preview) — swap them in the recipe if +# you serve equivalents elsewhere. Only the judge URLs (and the Tau2 user-sim +# model_id) come from here; they're config, not secrets, so no export needed (only +# INFERENCE_API_KEY is). nemo-skills uses the /v1 base; tau2-bench needs the full +# /v1/chat/completions. + +# HLE + AA-LCR + AA-Omniscience judges (ns_hle_aa, ns_aa_lcr, ns_omniscience) — shared inference host +# INFERENCE_JUDGE_URL=https:///v1 + +# Tau2 (tau2_bench_telecom) — judger + user-simulator model_ids are hardcoded in +# the recipe; only the shared endpoint URL comes from here # TAU2_ENDPOINT_URL=https:///v1/chat/completions # user + judger # terminal-bench-hard (AWS sandbox) diff --git a/.agents/skills/evaluation/recipes/examples/example_eval.yaml b/.agents/skills/evaluation/recipes/examples/example_eval.yaml index 1810a2de3d5..8f4182b67e4 100644 --- a/.agents/skills/evaluation/recipes/examples/example_eval.yaml +++ b/.agents/skills/evaluation/recipes/examples/example_eval.yaml @@ -46,6 +46,8 @@ defaults: - execution: slurm/default - deployment: vllm - _self_ +cluster: + sbatch_comment: '{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"480","reason":"benchmarking","description":"Eval benchmark low GPU utilization"}}' execution: hostname: ??? username: ${oc.env:USER} diff --git a/.agents/skills/evaluation/recipes/tasks/aa/hle.md b/.agents/skills/evaluation/recipes/tasks/aa/hle.md index 713a1fe1fbc..7acf29084d2 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa/hle.md +++ b/.agents/skills/evaluation/recipes/tasks/aa/hle.md @@ -6,12 +6,12 @@ ## Params -Text-only HLE, params aligned to Artificial Analysis Index v2; judge-scored. -Substitute the judge `model_id`/`url` with the literal values you keep in `.env` -(`HLE_JUDGE_MODEL_ID` rec. **GPT-4o**, `NS_JUDGE_URL`; see `recipes/env.example`) — -they're config, not secrets, so they don't need exporting. Only `api_key` -(`INFERENCE_API_KEY`) is exported and read by the harness. Keep the judge fixed -across comparable runs. +Text-only HLE, params aligned to Artificial Analysis Index v2; judge-scored. The +judge `model_id` is hardcoded in the fragment below (**GPT-4o**) — swap it for an +equivalent on your own endpoint if needed. The judge `url` still comes from `.env` +(`INFERENCE_JUDGE_URL`; see `recipes/env.example`) — config, not a secret, so no export. +Only `api_key` (`INFERENCE_API_KEY`) is exported and read by the harness. Keep the +judge fixed across comparable runs. `hle_strict_judge: true` (inside the `judge` block) enables strict judging. @@ -35,8 +35,8 @@ Use this inside the top-level `evaluation.tasks` list: params: extra: judge: - model_id: # from .env; recommended GPT-4o - url: # from .env (/v1 base) + model_id: azure/openai/gpt-4o # GPT-4o; use an equivalent on your own endpoint if needed + url: # from .env (/v1 base) api_key: INFERENCE_API_KEY # env-var name; exported, read by harness hle_strict_judge: true ``` diff --git a/.agents/skills/evaluation/recipes/tasks/aa/lcr.md b/.agents/skills/evaluation/recipes/tasks/aa/lcr.md index 76be02b3511..9a5fc973345 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa/lcr.md +++ b/.agents/skills/evaluation/recipes/tasks/aa/lcr.md @@ -6,10 +6,11 @@ ## Params -Judge-scored (equality checker). Substitute the judge `model_id`/`url` with the -literal values you keep in `.env` (`LCR_JUDGE_MODEL_ID` rec. **Qwen3 235B**, -`NS_JUDGE_URL`; see `recipes/env.example`) — config, not secrets, so no export -needed; only `api_key` (`INFERENCE_API_KEY`) is exported. Keep the judge fixed. +Judge-scored (equality checker). The judge `model_id` is hardcoded in the fragment +below (**Qwen3 235B**) — swap it for an equivalent on your own endpoint if needed. +The judge `url` still comes from `.env` (`INFERENCE_JUDGE_URL`; see `recipes/env.example`) +— config, not a secret, so no export; only `api_key` (`INFERENCE_API_KEY`) is +exported. Keep the judge fixed. AA-LCR needs long context: plan for roughly 120K input tokens plus 16K generation tokens. Set deployment `--max-model-len` to at least `131072`, and @@ -55,8 +56,8 @@ block. Per SKILL.md Step 3, the deployment flag must live inside extra: num_repeats: 16 judge: - model_id: # from .env; recommended Qwen3 235B - url: # from .env (/v1 base) + model_id: nvidia/qwen/qwen-235b # Qwen3 235B; use an equivalent on your own endpoint if needed + url: # from .env (/v1 base) api_key: INFERENCE_API_KEY # env-var name; exported, read by harness ``` diff --git a/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md b/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md new file mode 100644 index 00000000000..99f82388817 --- /dev/null +++ b/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md @@ -0,0 +1,44 @@ +# AA-Omniscience + +## Task Details + +- Reference: + +## Params + +Knowledge / hallucination benchmark, params aligned to Artificial Analysis Index +v2; judge-scored. The judge `model_id` is hardcoded in the fragment below +(**gcp/google/gemini-3-flash-preview**) — swap it for an equivalent on your own +endpoint if needed. The judge `url` comes from `.env` (`INFERENCE_JUDGE_URL`); only +`api_key` (`INFERENCE_API_KEY`) is exported and read by the harness. Keep the judge +fixed across comparable runs. + +`++parse_reasoning=False` is required (golden knob — omniscience scores the final +answer, not the reasoning trace). + +## YAML Fragment + +Use this inside the top-level `evaluation.tasks` list: + +```yaml +- name: nemo_skills.ns_omniscience + container: nvcr.io/nvidia/eval-factory/nemo-skills:26.05.1 + env_vars: + INFERENCE_API_KEY: host:INFERENCE_API_KEY + nemo_evaluator_config: + config: + params: + extra: + num_repeats: 10 + args: "++parse_reasoning=False" + judge: + api_key: INFERENCE_API_KEY + model_id: gcp/google/gemini-3-flash-preview # use an equivalent on your own endpoint if needed + url: # from .env (/v1 base) +``` + +## Score Extraction from mlflow + +Result (0-100): `omniscience_pass_at_1_avg-of-N_judge_correct` + +N is the repeat count (10). If the repeat count is unknown, use the highest available `avg-of-N`. diff --git a/.agents/skills/evaluation/recipes/tasks/aa/tau2_bench_telecom.md b/.agents/skills/evaluation/recipes/tasks/aa/tau2_bench_telecom.md index baae3e0cc62..edb0f42c184 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa/tau2_bench_telecom.md +++ b/.agents/skills/evaluation/recipes/tasks/aa/tau2_bench_telecom.md @@ -7,12 +7,13 @@ ## Params Tau2 uses the evaluated model as the agent plus a separate user-simulator endpoint; -keep both fixed across runs. Substitute the user-sim & judger `model_id`/`url` with the -literal values you keep in `.env` (`TAU2_USER_MODEL_ID` rec. **Qwen3 235B**, -`TAU2_JUDGER_MODEL_ID` rec. **gpt-oss-120B**, `TAU2_ENDPOINT_URL`; see -`recipes/env.example`) — config, not secrets, so no export needed; only `api_key` -(`INFERENCE_API_KEY`) is exported. tau2-bench needs the full `/v1/chat/completions` -URL (nemo-skills judges use the `/v1` base). +keep both fixed across runs. The judger (**gpt-oss-120B**) and user-simulator +(**Qwen3 235B**) `model_id`s are hardcoded in the fragment below — swap them for +equivalents on your own endpoint if needed. Only the shared `url` +(`TAU2_ENDPOINT_URL`) comes from `.env` (see `recipes/env.example`) — config, not a +secret, so no export needed; only `api_key` (`INFERENCE_API_KEY`) is exported. +tau2-bench needs the full `/v1/chat/completions` URL (nemo-skills judges use the +`/v1` base). For parallelism, we have to throttle to a smaller cap due to the test may be throttled by user and judger API rate limit. If frequent 429 errors are hit, the reported scores could be much lower. @@ -56,11 +57,11 @@ Use this inside the top-level `evaluation.tasks` list: skip_failed_samples: true n_samples: 8 user: - model_id: # from .env; recommended Qwen3 235B + model_id: nvidia/qwen/qwen-235b # Qwen3 235B; use an equivalent on your own endpoint if needed url: # from .env (full /v1/chat/completions) api_key: INFERENCE_API_KEY # env-var name; exported, read by harness judger: - model_id: # from .env; recommended gpt-oss-120B + model_id: nvidia/openai/gpt-oss-120b # gpt-oss-120B; use an equivalent on your own endpoint if needed url: # from .env (full /v1/chat/completions) api_key: INFERENCE_API_KEY # env-var name; exported, read by harness ``` From e5861d634620ef7c25eda00bb8ccde33e907bb58 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Fri, 26 Jun 2026 17:14:31 +0000 Subject: [PATCH 2/4] Address PR #1834 review comments - references/quantization-benchmarks.md: add AA-Omniscience to the task table, the AA text-suite recommended set, the repeat-count note (num_repeats=10), and the judge-endpoint note, so it stays in sync with the AA suite list in SKILL.md (meenchen). - env.example: trim the judge-endpoint note to a brief pointer (model_ids live in each recipe) and drop the stale claim that the Tau2 user-sim model_id comes from .env -- it is hardcoded in the recipe (chadvoegele, coderabbit). - SKILL.md: clarify that the re-submittable run.sub is the one NEL generated in the job's run dir (chadvoegele). Signed-off-by: Chenjie Luo --- .agents/skills/evaluation/SKILL.md | 2 +- .agents/skills/evaluation/recipes/env.example | 11 ++++------- .../references/quantization-benchmarks.md | 13 ++++++++----- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/.agents/skills/evaluation/SKILL.md b/.agents/skills/evaluation/SKILL.md index 5b36002ec8e..94b4b392f41 100644 --- a/.agents/skills/evaluation/SKILL.md +++ b/.agents/skills/evaluation/SKILL.md @@ -240,7 +240,7 @@ On SLURM, several deploy/eval failures are invisible to `--dry-run` and only sur Evals that exceed 4h of wall-clock time are handled by **NEL's built-in dependency-chain resume**, not by shrinking the eval. NEL submits the first SLURM job; if it hits walltime, a dependent follow-on job resumes from the response/result caches the first job wrote, then queues another follow-on. Long evals continue across walltime windows automatically. See `references/run-validation.md#nel-timeout-and-resume-behavior` for the full mechanism. -**Preemption / external kill — resume manually with `sbatch run.sub`.** On a preemptible account (common on busy internal clusters) the scheduler can **CANCEL** a run mid-eval for a higher-priority job — `sacct -j ` shows `CANCELLED by ` (a `svc-*` service account) with `Elapsed` well under the 4h walltime. NEL does **not** auto-resume this (its dependency chain only fires on a genuine walltime timeout). But the job's generated `run.sub` is **re-submittable** and resumes from the same `output_dir` + response cache (`skip_filled`), continuing from the partial output rather than restarting: +**Preemption / external kill — resume manually with `sbatch run.sub`.** On a preemptible account (common on busy internal clusters) the scheduler can **CANCEL** a run mid-eval for a higher-priority job — `sacct -j ` shows `CANCELLED by ` (a `svc-*` service account) with `Elapsed` well under the 4h walltime. NEL does **not** auto-resume this (its dependency chain only fires on a genuine walltime timeout). But the `run.sub` that NEL generated for the job (in its run dir) is **re-submittable** and resumes from the same `output_dir` + response cache (`skip_filled`), continuing from the partial output rather than restarting: ```bash ssh "cd /-// && sbatch run.sub" diff --git a/.agents/skills/evaluation/recipes/env.example b/.agents/skills/evaluation/recipes/env.example index edf6d38eccd..876993513ed 100644 --- a/.agents/skills/evaluation/recipes/env.example +++ b/.agents/skills/evaluation/recipes/env.example @@ -35,13 +35,10 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # --- Optional: judge / user-simulator endpoints (URL only) --- # -# Judge `model_id`s are now HARDCODED in the task recipes (HLE azure/openai/gpt-4o, -# AA-LCR nvidia/qwen/qwen-235b, Tau2 judger nvidia/openai/gpt-oss-120b, -# AA-Omniscience gcp/google/gemini-3-flash-preview) — swap them in the recipe if -# you serve equivalents elsewhere. Only the judge URLs (and the Tau2 user-sim -# model_id) come from here; they're config, not secrets, so no export needed (only -# INFERENCE_API_KEY is). nemo-skills uses the /v1 base; tau2-bench needs the full -# /v1/chat/completions. +# Judge / user-simulator `model_id`s are hardcoded in each task recipe (swap them +# there if you serve an equivalent). Only the endpoint URLs come from here — config, +# not secrets, so no export needed (only INFERENCE_API_KEY is). nemo-skills uses the +# /v1 base; tau2-bench needs the full /v1/chat/completions. # HLE + AA-LCR + AA-Omniscience judges (ns_hle_aa, ns_aa_lcr, ns_omniscience) — shared inference host # INFERENCE_JUDGE_URL=https:///v1 diff --git a/.agents/skills/evaluation/references/quantization-benchmarks.md b/.agents/skills/evaluation/references/quantization-benchmarks.md index f462b53ebec..9b39fac806f 100644 --- a/.agents/skills/evaluation/references/quantization-benchmarks.md +++ b/.agents/skills/evaluation/references/quantization-benchmarks.md @@ -27,6 +27,7 @@ to precision loss. The Artificial Analysis (AA) Index v2 suite under | `tasks/aa/ifbench.md` | IFBench | Instruction following | Low — format-compliance is robust; even aggressive FP4 usually shows only small drops | | `tasks/aa/mmmu_pro.md` | MMMU-Pro | Multimodal reasoning | VLM-only; usually Low/Medium when only the LLM is quantized (vision encoder/adapter typically stay BF16) | | `tasks/aa/tau2_bench_telecom.md` | Tau2-Bench Telecom | Agentic tool use (user-simulator + judge) | Medium-high — tool-call JSON is brittle, but user-sim + judge variance often dominates the signal | +| `tasks/aa/omniscience.md` | AA-Omniscience | Knowledge reliability (`ns_omniscience`, nemo-skills, `num_repeats: 10`) — correct vs hallucinate vs abstain on obscure facts, judge-scored | Medium — measures the hallucination/abstention balance; aggressive precision loss can erode factual recall and shift the omni-index | ## Recommended sets by use case @@ -34,7 +35,7 @@ to precision loss. The Artificial Analysis (AA) Index v2 suite under |----------|-----------| | Quick sanity check | GPQA | | Standard quant validation (text LLM) | GPQA, SciCode, LCR | -| AA / Artificial Analysis suite (text LLM) | All `tasks/aa/` text tasks: GPQA, HLE, LCR, SciCode, IFBench, Tau2-Bench Telecom | +| AA / Artificial Analysis suite (text LLM) | All `tasks/aa/` text tasks: GPQA, HLE, LCR, SciCode, IFBench, Tau2-Bench Telecom, AA-Omniscience | | AA / Artificial Analysis suite (multimodal) | AA text suite + MMMU-Pro | | Code-focused model | LiveCodeBench, SciCode | | Reasoning model | AIME 2025, GPQA, HLE | @@ -53,10 +54,12 @@ to precision loss. The Artificial Analysis (AA) Index v2 suite under do **not** lower them for quant comparisons, or noise will mask real regressions. The field name differs by harness: `n_samples` for simple-evals (AIME `64`) and tau2-bench (Tau2 `8`); `num_repeats` for nemo-skills - (AA-LCR/GPQA `16`, LiveCodeBench/SciCode `8`, IFBench `5`, MMLU-Pro `1`). -- **Judge / user-simulator endpoints** are required by AA-LCR, HLE AA, and - Tau2-Bench Telecom. Keep the judge and (for Tau2) user-simulator models - fixed across baseline and quantized runs for apples-to-apples comparison. + (AA-LCR/GPQA `16`, AA-Omniscience `10`, LiveCodeBench/SciCode `8`, IFBench `5`, + MMLU-Pro `1`). +- **Judge / user-simulator endpoints** are required by AA-LCR, HLE AA, + AA-Omniscience, and Tau2-Bench Telecom. Keep the judge and (for Tau2) + user-simulator models fixed across baseline and quantized runs for + apples-to-apples comparison. - **IFBench** is the least quant-sensitive in the set but still useful as a regression check for aggressive formats (NVFP4, INT4-AWQ). From ac56f2c1493a613b88a5ebe2308d96e9b0badf90 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Fri, 26 Jun 2026 17:17:19 +0000 Subject: [PATCH 3/4] Report Omniscience Index + non-hallucination rate in recipe Per review (jinhangchoi): AA-Omniscience headline is the hallucination-adjusted Omniscience Index, not just accuracy. Expand the score-extraction section to list accuracy (judge_correct), the Omniscience Index (judge_omni_index), and the non-hallucination rate (1 - judge_omni_hallucination), with the AA methodology link. Signed-off-by: Chenjie Luo --- .../skills/evaluation/recipes/tasks/aa/omniscience.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md b/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md index 99f82388817..b26f34b65d6 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md +++ b/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md @@ -39,6 +39,11 @@ Use this inside the top-level `evaluation.tasks` list: ## Score Extraction from mlflow -Result (0-100): `omniscience_pass_at_1_avg-of-N_judge_correct` +AA-Omniscience reports a few related numbers (see the AA methodology: +): -N is the repeat count (10). If the repeat count is unknown, use the highest available `avg-of-N`. +- **Accuracy** (0-100): `omniscience_pass_at_1_avg-of-N_judge_correct` — % of questions answered correctly. +- **Omniscience Index** (-100 to 100): `omniscience_pass_at_1_avg-of-N_judge_omni_index` — accuracy net of hallucinations; AA's headline metric, which rewards abstaining over guessing wrong (so it can be negative). +- **Non-hallucination rate** (0-100): `100 - omniscience_pass_at_1_avg-of-N_judge_omni_hallucination` — the `judge_omni_hallucination` key is the hallucination rate, so non-hallucination = `1 - hallucination`. + +N is the repeat count (10). If the repeat count is unknown, use the highest available `avg-of-N`. From 07cc217463b9425e6bca541428ca4038c96d28db Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Fri, 26 Jun 2026 17:25:31 +0000 Subject: [PATCH 4/4] Make Omniscience Index the primary reported metric Per review: AA-Omniscience's headline is the hallucination-adjusted Omniscience Index, so report it as the primary result; accuracy and non-hallucination rate become secondary. Signed-off-by: Chenjie Luo --- .agents/skills/evaluation/recipes/tasks/aa/omniscience.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md b/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md index b26f34b65d6..f80d7873757 100644 --- a/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md +++ b/.agents/skills/evaluation/recipes/tasks/aa/omniscience.md @@ -39,11 +39,11 @@ Use this inside the top-level `evaluation.tasks` list: ## Score Extraction from mlflow -AA-Omniscience reports a few related numbers (see the AA methodology: -): +**Primary result — Omniscience Index** (-100 to 100): `omniscience_pass_at_1_avg-of-N_judge_omni_index`. This is AA's headline metric: accuracy net of hallucinations, rewarding abstention over guessing wrong (so it can be negative). Report this one. See the AA methodology: . + +Also report (same `pass_at_1_avg-of-N` aggregation): - **Accuracy** (0-100): `omniscience_pass_at_1_avg-of-N_judge_correct` — % of questions answered correctly. -- **Omniscience Index** (-100 to 100): `omniscience_pass_at_1_avg-of-N_judge_omni_index` — accuracy net of hallucinations; AA's headline metric, which rewards abstaining over guessing wrong (so it can be negative). - **Non-hallucination rate** (0-100): `100 - omniscience_pass_at_1_avg-of-N_judge_omni_hallucination` — the `judge_omni_hallucination` key is the hallucination rate, so non-hallucination = `1 - hallucination`. N is the repeat count (10). If the repeat count is unknown, use the highest available `avg-of-N`.