diff --git a/docs/evaluator/agent-eval/evaluate-deployed-agent.mdx b/docs/evaluator/agent-eval/evaluate-deployed-agent.mdx
new file mode 100644
index 0000000000..e974fca3eb
--- /dev/null
+++ b/docs/evaluator/agent-eval/evaluate-deployed-agent.mdx
@@ -0,0 +1,210 @@
+---
+title: "Evaluate a Deployed Agent over HTTP"
+description: "Point NeMo Evaluator at an agent reachable over HTTP by describing it as a GenericAgent target — the request to send and where the answer is in the response — then score it exactly like the quickstart."
+---
+
+The [quickstart](/documentation/evaluate-models/agent-eval/quickstart) evaluated an in-process agent.
+This guide evaluates an agent that runs **behind an HTTP endpoint** — a service you (or someone else)
+deployed. Everything else is the same: the same tasks, the same metrics, the same `run()`. Only the
+**target** changes.
+
+You describe an HTTP agent as a `GenericAgent`: its URL, the JSON request to send, and where the
+answer lives in the response. The evaluator then calls that endpoint over real HTTP for each task.
+
+
+
+This guide assumes you've done the [quickstart](/documentation/evaluate-models/agent-eval/quickstart)
+(install, tasks, metrics, `run()`). It reuses that quickstart's `KeywordMatchMetric` and tasks.
+
+
+
+## 1. Describe the agent
+
+A `GenericAgent` has three parts you configure:
+
+- **`url`** — where the agent is reachable.
+- **`body`** — the JSON request to POST, as a Jinja template rendered against the task's `inputs`.
+ Reference them directly: the task's instruction is `{{ instruction }}`. The payload is entirely
+ yours — the evaluator sends exactly what `body` produces.
+- **`response_path`** — a JSONPath into the response that selects the agent's answer.
+
+```python
+from nemo_evaluator_sdk.enums import AgentFormat
+from nemo_evaluator_sdk.values import GenericAgent
+
+agent = GenericAgent(
+ name="my-http-agent",
+ url="http://127.0.0.1:8000/agent",
+ format=AgentFormat.GENERIC,
+ body={"message": "{{ instruction }}"}, # -> {"message": "What is the capital of France?"}
+ response_path="$.answer", # reads {"answer": "..."} from the response
+ # api_key_secret="MY_AGENT_TOKEN", # optional: env var (local run) or platform secret (job), sent as a bearer token
+)
+```
+
+Set `body` and `response_path` to match **your** agent's request and response shape. Uncomment
+`api_key_secret` for an agent that needs a token — for a local `run()` it names an environment
+variable in your process, and for a submitted job it names a platform secret in the workspace; either
+way its value is sent as a bearer token on each request. The local stand-in below needs no auth, so
+this walkthrough leaves it off.
+
+## 2. Run it against a local stand-in
+
+To try this end to end without deploying anything, stand up a tiny local HTTP server that plays the
+role of the agent. In practice you'd skip this and point `url` at your real endpoint.
+
+```python
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+
+class _AgentHandler(BaseHTTPRequestHandler):
+ def do_POST(self) -> None:
+ payload = json.loads(self.rfile.read(int(self.headers.get("Content-Length", "0"))) or b"{}")
+ text = str(payload.get("message", "")).lower()
+ answer = "Paris" if "france" in text else "Tokyo" if "japan" in text else "unsure"
+ body = json.dumps({"answer": answer}).encode()
+ self.send_response(200)
+ self.send_header("Content-Type", "application/json")
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *args: object) -> None:
+ return # keep the demo output quiet
+```
+
+## 3. Run the evaluation
+
+With an agent target and no inference function supplied, `run()` generates answers by making real HTTP
+calls to the agent's `url`.
+
+```python
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
+
+result = await AgentEvaluator().run(
+ tasks=make_tasks(), # from the quickstart
+ target=agent,
+ config=AgentEvalRunConfig(parallelism=2),
+)
+
+for aggregate in result.summary.scores.scores:
+ print(f"{aggregate.name}: {aggregate.mean}")
+```
+
+## Point at your real agent
+
+To evaluate your own deployed agent, change three things and drop the local server:
+
+1. **`url`** → your agent's endpoint.
+2. **`body`** → the request your agent expects (reference the task's inputs, e.g. `{{ instruction }}`).
+3. **`response_path`** → the JSONPath to the answer in your agent's response.
+
+Add `api_key_secret="MY_AGENT_TOKEN"` if it needs auth (an env var for a local run, a platform secret for a job). If your agent is a **NeMo Agent Toolkit**
+workflow, use `NemoAgentToolkitAgent` instead of `GenericAgent` — it targets a NAT endpoint's fixed
+protocol, so you don't hand-write a `body` template.
+
+## Full script
+
+```python
+import asyncio
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
+from nemo_evaluator_sdk.enums import AgentFormat
+from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
+from nemo_evaluator_sdk.values import GenericAgent
+
+
+class KeywordMatchMetric:
+ @property
+ def type(self) -> str:
+ return "keyword_match"
+
+ def output_spec(self) -> list[MetricOutputSpec]:
+ return [MetricOutputSpec.continuous_score("score")]
+
+ async def compute_scores(self, input: MetricInput) -> MetricResult:
+ expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
+ answer = (input.candidate.output_text or "").lower()
+ return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
+
+
+def make_tasks() -> list[AgentEvalTask]:
+ return [
+ AgentEvalTask(id="capital-france", intent="Answer the geography question.",
+ inputs={"instruction": "What is the capital of France?"},
+ reference={"expected": "Paris"}, metrics=[KeywordMatchMetric()]),
+ AgentEvalTask(id="capital-japan", intent="Answer the geography question.",
+ inputs={"instruction": "What is the capital of Japan?"},
+ reference={"expected": "Tokyo"}, metrics=[KeywordMatchMetric()]),
+ ]
+
+
+# A stand-in "deployed agent": a tiny local HTTP server. Replace with your real agent's URL.
+class _AgentHandler(BaseHTTPRequestHandler):
+ def do_POST(self) -> None:
+ payload = json.loads(self.rfile.read(int(self.headers.get("Content-Length", "0"))) or b"{}")
+ text = str(payload.get("message", "")).lower()
+ answer = "Paris" if "france" in text else "Tokyo" if "japan" in text else "unsure"
+ body = json.dumps({"answer": answer}).encode()
+ self.send_response(200)
+ self.send_header("Content-Type", "application/json")
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *args: object) -> None:
+ return
+
+
+async def main() -> None:
+ httpd = ThreadingHTTPServer(("127.0.0.1", 0), _AgentHandler)
+ threading.Thread(target=httpd.serve_forever, daemon=True).start()
+ host, port = httpd.server_address
+
+ agent = GenericAgent(
+ name="my-http-agent",
+ url=f"http://{host}:{port}/agent",
+ format=AgentFormat.GENERIC,
+ body={"message": "{{ instruction }}"},
+ response_path="$.answer",
+ )
+ try:
+ result = await AgentEvaluator().run(
+ tasks=make_tasks(),
+ target=agent,
+ config=AgentEvalRunConfig(parallelism=2),
+ )
+ finally:
+ httpd.shutdown()
+
+ for aggregate in result.summary.scores.scores:
+ print(f"{aggregate.name}: {aggregate.mean}")
+ for trial in result.trials:
+ print(f" {trial.task_id}: {trial.output.output_text!r}")
+
+
+asyncio.run(main())
+```
+
+Expected output — the answers came back over HTTP and both contain the expected keyword:
+
+```
+keyword_match.score: 1.0
+ capital-france: 'Paris'
+ capital-japan: 'Tokyo'
+```
+
+## Next steps
+
+
+
+
+
+
+- Score the agent's **trajectory** (its tool use), not just the final answer.
+- Give a task **multiple metrics** and combine them into a named **view**.
diff --git a/docs/evaluator/agent-eval/harbor-runner.mdx b/docs/evaluator/agent-eval/harbor-runner.mdx
new file mode 100644
index 0000000000..cff1a1814b
--- /dev/null
+++ b/docs/evaluator/agent-eval/harbor-runner.mdx
@@ -0,0 +1,167 @@
+---
+title: "Evaluate a Harbor Task Suite"
+description: "Run an existing Harbor task dataset through NeMo Evaluator's Harbor runner — Harbor executes each task in a Docker sandbox and its verifier emits a reward, which the SDK scores and reports like any other agent-eval run."
+---
+
+[Harbor](https://www.harborframework.com) is a container-based harness for agentic tasks: it runs each
+task in a Docker sandbox, lets an agent work in it, then runs a **verifier** that emits a **reward**
+(see Harbor's [Core Concepts](https://www.harborframework.com/docs/core-concepts) for its task, trial,
+and job model). If you already have Harbor task datasets, the **Harbor runner** runs them and scores
+the verifier reward through agent-eval — the same
+[`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the same result and bundle as the
+[quickstart](/documentation/evaluate-models/agent-eval/quickstart). Only the runner changes.
+
+
+
+Unlike the quickstart, this runner is **not** zero-dependency — it shells out to Harbor and Docker:
+
+- **Python ≥ 3.12**
+- **Docker** installed and running
+- **Harbor**, installed separately: `uv pip install "harbor>=0.16.1"`. Harbor is intentionally **not**
+ a dependency of `nemo-platform[nemo-evaluator-sdk]`, so the rest of the SDK stays lightweight.
+
+The runner raises a clear error pointing at this install step if `harbor` is missing.
+
+
+
+## The dataset
+
+A Harbor dataset is a directory of **task folders**. For discovery, the runner needs two files per
+task — the rest of the task format (environment, verifier, and solution config) is Harbor's own:
+
+```
+my-suite/
+ hello-world/
+ task.toml # [task] name = "harbor/hello-world"
+ instruction.md # Create a file called hello.txt with "Hello, world!" as the content.
+ ...
+```
+
+`discover_harbor_tasks` reads each folder into an `AgentEvalTask`: the `[task] name` becomes the task
+`id` and its human-readable `intent`, and `instruction.md` becomes `inputs["instruction"]` — the
+instruction the agent is prompted with. See
+[Harbor's task documentation](https://www.harborframework.com/docs/tasks) for the full task format.
+
+
+
+The repo ships a one-task
+[example dataset](https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/packages/nemo_evaluator_sdk/examples/harbor/hello_world_dataset)
+you can clone and point at (it is not shipped in the installed wheel). Or point the dataset path at
+your own Harbor suite.
+
+
+
+## Run it
+
+A Harbor run is a normal agent-eval run: `AgentEvaluator().run(tasks=..., target=runner)`, exactly like
+the [quickstart](/documentation/evaluate-models/agent-eval/quickstart) (a callable) and the
+[deployed-agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) guide (an HTTP
+target). Here the target is a `HarborAgentTaskRunner`.
+
+```python
+import asyncio
+from pathlib import Path
+
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import (
+ HarborAgentTaskRunner,
+ HarborRuntimeConfig,
+ discover_harbor_tasks,
+)
+
+
+async def main() -> None:
+ # INPUT: your Harbor task suite (a directory of task folders).
+ tasks = discover_harbor_tasks("path/to/my-suite")
+
+ runner = HarborAgentTaskRunner(
+ config=HarborRuntimeConfig(
+ # OUTPUT: where Harbor writes its / results tree (not the dataset).
+ jobs_dir=Path("./harbor-jobs"),
+ agent_name="oracle", # a built-in Harbor agent — see "Choosing an agent"
+ )
+ )
+
+ # Same AgentEvaluator as every agent-eval run; only the target changes.
+ result = await AgentEvaluator().run(tasks=tasks, target=runner)
+
+ for aggregate in result.summary.scores.scores:
+ print(f"{aggregate.name}: {aggregate.mean}")
+
+
+asyncio.run(main())
+```
+
+The three pieces map onto the model: `discover_harbor_tasks` turns the suite into tasks,
+`HarborAgentTaskRunner` runs the Harbor job and returns one trial per Harbor trial, and `AgentEvaluator`
+scores each trial's reward with `HarborRewardMetric`.
+
+
+
+**`dataset_path` is input; `jobs_dir` is output.** The dataset is your read-only task suite. `jobs_dir`
+is a directory the runner writes into — Harbor's per-trial results land under `jobs_dir//`,
+and that directory doubles as a re-run cache (see [below](#attempts-concurrency-and-caching)).
+
+
+
+The `oracle` agent is Harbor's reference agent — it produces a passing trial on a well-formed task —
+so for the one-task example above the verifier reward is `1.0`:
+
+```
+harbor_reward.reward: 1.0
+```
+
+Swap `agent_name` (or `agent_import_path`) for your own agent to get a real score.
+
+## Choosing an agent
+
+`HarborRuntimeConfig` decides what runs inside each sandbox (see
+[Harbor's agent documentation](https://www.harborframework.com/docs/agents)):
+
+- **`agent_name`** — a built-in Harbor agent (for example `"oracle"`, Harbor's reference agent, handy
+ as a smoke test that the harness and dataset are wired up).
+- **`agent_import_path`** — your own Harbor agent, e.g. `"my_agent_module:MyAgent"`. Set `agent_dir`
+ as well when it's a loose file rather than an installed package. Overrides `agent_name`.
+- **`agent_model_name`** — the model slug handed to the agent.
+
+## How scoring works
+
+1. Harbor runs each task in its sandbox and writes a `__/result.json` per trial, including
+ the reward from the task's [verifier](https://www.harborframework.com/docs/tasks).
+2. The runner reads that reward onto each trial's metadata, and `HarborRewardMetric` scores it — the
+ `reward` value, or `0.0` for a trial whose verifier emitted none.
+3. `result.summary` aggregates the reward across tasks (`harbor_reward.reward`), `result.trials` holds
+ each trial's status and evidence, and — if you pass a `config` with an `output_dir` to `run()` — the
+ run **bundle** (including `report.html`) is written like any other agent-eval run.
+
+## Attempts, concurrency, and caching
+
+- **`n_attempts`** — trials Harbor runs per task; **`n_concurrent_trials`** — how many run at once.
+- The job directory doubles as a **cache**. Pin a stable `job_name` and a completed job whose results
+ already cover every requested task is re-scored instead of re-run. The default timestamped `job_name`
+ always runs fresh; set `force_rerun=True` to delete an existing job dir first.
+
+## Shortcut: `run_harbor_eval`
+
+When a run is exactly "one Harbor suite, scored by its reward," `run_harbor_eval` collapses the three
+steps above — discover, run, score — into a single call:
+
+```python
+from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
+
+result = await run_harbor_eval(
+ HarborRuntimeConfig(jobs_dir="./harbor-jobs", agent_name="oracle"),
+ dataset_path="path/to/my-suite",
+)
+```
+
+It uses the same `AgentEvaluator` and `HarborRewardMetric` under the hood. Prefer the explicit form
+above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across
+targets.
+
+## Next steps
+
+
+
+
+
diff --git a/docs/evaluator/agent-eval/index.mdx b/docs/evaluator/agent-eval/index.mdx
new file mode 100644
index 0000000000..e561c8796c
--- /dev/null
+++ b/docs/evaluator/agent-eval/index.mdx
@@ -0,0 +1,119 @@
+---
+title: "Agent Evaluation"
+description: "The task-driven evaluation model — define tasks, run an agent to produce trials, and score each trial (output and trajectory) with per-task metrics."
+---
+
+
+
+Agent evaluation is the **task-driven** shape of NeMo Evaluator (see
+[Dataset-Driven vs Task-Driven Evaluation](/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation)
+for how it compares to metrics). You define **tasks**, an **agent** performs each one,
+and you score the resulting **trial** — not just whether the final answer is right, but *how the agent
+got there*. Each task carries its own metrics, so a single suite can grade heterogeneous work.
+
+## The model
+
+One run flows through five pieces:
+
+**Task → (runner) → Trial → (metrics) → Scores → Result**
+
+- **Task** (`AgentEvalTask`) — the unit of work. Its fields:
+ - `intent` — a human-readable description of the goal; metadata for the suite's authors, **not** shown to the agent.
+ - `inputs` — the instruction and anything the agent starts from (what the agent actually sees).
+ - `reference` (optional) — grader-only, held-out ground truth the agent never sees.
+ - `metrics` — the scorers for **this** task.
+ - `views` (optional) — combine several of the task's metric outputs into one named, reported score (see *Score by component* under [Key properties](#key-properties)).
+- **Runner** (`AgentTaskRunner`) — performs each task and returns trials through one small interface,
+ `run_tasks(tasks) -> [AgentEvalTrial]`. The SDK ships runners for a plain async callable, a deployed
+ agent over HTTP, and container/harness backends; you can supply your own. Scoring never depends on
+ which runner produced a trial.
+- **Trial** (`AgentEvalTrial`) — the durable record of one attempt: the agent's final `output`, the
+ `evidence` it produced (an [Agent Trajectory Interchange Format (ATIF)](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trace of steps and tool calls, final filesystem state, logs), a
+ `status` (`completed` / `partial` / `failed`), and arbitrary metadata. The trial — not the runner — is what
+ gets scored, and it can be re-scored offline later.
+- **Metrics** (`Metric`) — each task's metrics score its trials through
+ `compute_scores(input) -> MetricResult`. A metric can read the final output **and** the evidence, so
+ it can grade the outcome (did it answer correctly?) or the trajectory (did it use the expected tool?).
+- **Result** (`AgentEvaluator` → `AgentEvalResult`) — the evaluator orchestrates the run and returns
+ the trials, per-trial scores, and an aggregated `summary`. Point it at an output directory and it
+ also writes a run **bundle**: `run.json`, `trials.jsonl`, `scores.jsonl`, `summary.json`, and a
+ browsable `report.html`.
+
+## A minimal example
+
+
+
+Conceptual — for a runnable, end-to-end version see the quickstart (coming in this section).
+
+
+
+```python
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask
+
+task = AgentEvalTask(
+ id="capital-france",
+ intent="Name the capital of France.",
+ inputs={"instruction": "What is the capital of France?"},
+ reference={"expected": "Paris"}, # grader-only; never shown to the agent
+ metrics=[OutcomeMetric()], # this task's own scorer(s)
+)
+
+# target is a Model, a deployed Agent (HTTP), or any AgentTaskRunner.
+result = await AgentEvaluator().run(tasks=[task], target=my_agent)
+print(result.summary)
+```
+
+## Key properties
+
+- **Runner-agnostic scoring.** The scorer only ever sees an `AgentEvalTrial`, so the same tasks and
+ metrics score any runner's output — or previously stored trials re-scored offline.
+- **Per-task metrics.** Metrics are attached to each task, not once for the whole run, so a suite can
+ grade heterogeneous work (docs, Q&A, tests, net-new code) each with the checks that fit it.
+- **Score by component.** A single run can score at several levels:
+ - the **task outcome** — the final answer;
+ - the **trajectory** — how the agent worked (its tool use and steps);
+ - **views** — named roll-ups you define on a task that combine two or more of its metric outputs into one reported score (for example, averaging an accuracy metric and a tool-use metric into a single `quality` score);
+ - the **run-level aggregate** — results also roll up across the whole run.
+- **Runs locally.** A full run — including the `report.html` dashboard — is produced on your machine
+ with no platform services required; the same run can also be submitted as a platform job.
+- **Measurement, not decisions.** The evaluator produces scores, aggregates, and provenance — it
+ doesn't decide pass/fail, gate a release, or compare runs. Those decisions belong to whatever
+ consumes the results.
+
+## Targets
+
+An evaluation target is what performs the tasks:
+
+- a **`Model`** — a chat/completions endpoint;
+- a deployed **`Agent`** reachable over HTTP — a `GenericAgent` (any JSON endpoint) or a
+ `NemoAgentToolkitAgent`;
+- a custom **runner** (`AgentTaskRunner`) — anything that can turn tasks into trials.
+
+See [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)
+for how to describe an HTTP agent target.
+
+## In this section
+
+- [Quickstart](/documentation/evaluate-models/agent-eval/quickstart) — evaluate an agent end to end in
+ a local process, no services required.
+- [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)
+ — point a run at an agent behind an HTTP endpoint.
+- [Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner) — run an
+ existing Harbor dataset in Docker and score its verifier reward.
+- [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) — score the
+ trajectory (not just the answer) and roll metrics into a named view.
+- [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners) — reference for
+ what a run can point at.
+- [Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics) — reference for the
+ `Metric` protocol.
+- [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) — what a run returns and
+ the on-disk bundle it writes.
+
+## Related
+
+
+
+
+
+
diff --git a/docs/evaluator/agent-eval/quickstart.mdx b/docs/evaluator/agent-eval/quickstart.mdx
new file mode 100644
index 0000000000..1e95bcae9d
--- /dev/null
+++ b/docs/evaluator/agent-eval/quickstart.mdx
@@ -0,0 +1,244 @@
+---
+title: "Agent Evaluation Quickstart"
+description: "Evaluate a simple agent end to end with the NeMo Evaluator SDK — define tasks and a metric, run locally, and read the scores and HTML report. No platform services or API keys required."
+---
+
+In this quickstart you'll evaluate an agent on two tasks, score its answers, and produce a run
+bundle with a browsable HTML report — all on your machine, with **no platform services and no API
+keys**. It takes about five minutes.
+
+For the concepts behind tasks, trials, runners, and metrics, see
+[Agent Evaluation](/documentation/evaluate-models/agent-eval). This page is the hands-on version.
+
+## Prerequisites
+
+- Python 3.11–3.14
+- The SDK:
+
+```bash
+pip install nemo-platform[nemo-evaluator-sdk]
+```
+
+
+
+Everything here runs in a single local Python process. You'll swap the stand-in agent for a real
+model or deployed agent once you've seen the flow.
+
+
+
+## 1. Define a metric
+
+A metric scores one trial. It implements three things: a `type` (its name), an `output_spec()` (the
+values it emits), and `compute_scores()` (the scoring logic). Here's a tiny one that scores `1.0`
+when the agent's answer contains an expected keyword and `0.0` otherwise. It reads the agent's answer
+from `input.candidate.output_text` and the task's grader-only `reference` from `input.row.data`.
+
+```python
+from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
+
+
+class KeywordMatchMetric:
+ """Score 1.0 when the agent's answer contains the expected keyword, else 0.0."""
+
+ @property
+ def type(self) -> str:
+ return "keyword_match"
+
+ def output_spec(self) -> list[MetricOutputSpec]:
+ return [MetricOutputSpec.continuous_score("score")]
+
+ async def compute_scores(self, input: MetricInput) -> MetricResult:
+ expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
+ answer = (input.candidate.output_text or "").lower()
+ return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
+```
+
+## 2. Define your tasks
+
+Each `AgentEvalTask` is one unit of work: an `intent`, the `inputs` the agent acts on (here, an
+`instruction`), a grader-only `reference` (held-out truth the agent never sees), and the `metrics`
+that score **this** task. The `intent` is a human-readable note about the task's goal — metadata for
+whoever reads or maintains the suite. It is **not** passed to the agent, which only ever sees
+`inputs`. Because metrics live on the task, different tasks can use different scorers.
+
+```python
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask
+
+
+def make_tasks() -> list[AgentEvalTask]:
+ return [
+ AgentEvalTask(
+ id="capital-france",
+ intent="Answer the geography question.",
+ inputs={"instruction": "What is the capital of France?"},
+ reference={"expected": "Paris"},
+ metrics=[KeywordMatchMetric()],
+ ),
+ AgentEvalTask(
+ id="capital-japan",
+ intent="Answer the geography question.",
+ inputs={"instruction": "What is the capital of Japan?"},
+ reference={"expected": "Tokyo"},
+ metrics=[KeywordMatchMetric()],
+ ),
+ ]
+```
+
+## 3. Provide an agent
+
+An agent is anything that turns a task into an answer. The simplest option is an async function; the
+SDK provides a runner — `CallableAgentTaskRunner` — that wraps your function so it plugs into a run.
+The runner hands your function the whole task and expects an answer back (here, a canned response
+keyed by task `id`). We'll use a stand-in so the quickstart runs with no external services; later
+you'll point the evaluation at a real model or a
+[deployed agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) instead.
+
+```python
+async def my_agent(task: AgentEvalTask) -> str:
+ # Stand-in agent — replace with a real model or deployed agent later.
+ answers = {
+ "capital-france": "The capital of France is Paris.",
+ "capital-japan": "The capital of Japan is Tokyo.",
+ }
+ return answers[task.id]
+```
+
+## 4. Run the evaluation
+
+`AgentEvaluator.run()` sends the tasks to the runner, collects the trials, scores them, and returns
+an `AgentEvalResult`. Setting `output_dir` also writes a run bundle to disk. (`run()` is async, so it
+lives inside an `async` function — see the full script below.)
+
+```python
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
+
+result = await AgentEvaluator().run(
+ tasks=make_tasks(),
+ target=CallableAgentTaskRunner(my_agent),
+ config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
+)
+
+for aggregate in result.summary.scores.scores:
+ print(f"{aggregate.name}: {aggregate.mean}")
+```
+
+## 5. Read the results
+
+`run()` returns an `AgentEvalResult` you can work with directly in Python:
+
+- `result.summary` — aggregated scores per metric output (the mean you just printed), plus coverage
+ counts (how many trials were scored, failed, or missing an output).
+- `result.scores` — one entry per (task, trial, metric): the metric outputs and their status.
+- `result.trials` — each trial: the agent's answer, the evidence it produced (trajectory, final
+ state, logs), and its status (`completed` / `partial` / `failed`).
+- `result.run_id` — a stable identifier for this run.
+
+Because you set `output_dir`, the same data was also written to `./agent-eval-run/` as a run bundle:
+
+| File | Contents |
+|---|---|
+| `summary.json` | aggregated scores per metric output (mean / min / max / std-dev / counts) and coverage |
+| `scores.jsonl` | one row per (task, trial, metric) — the metric outputs, status, and any diagnostics |
+| `trials.jsonl` | one row per trial — the agent's output, its evidence, and status |
+| `tasks.jsonl` | the tasks that were evaluated |
+| `run.json` | the run manifest — the run id and a map of the artifact files |
+| `benchmark.json` | benchmark-grouping metadata for the run |
+| `report.html` | a browsable dashboard of the run — open it in a browser |
+
+The in-memory result and the on-disk bundle hold the same information: use the result object for
+programmatic follow-up, and the bundle (especially `report.html`) to inspect or share a run.
+
+## Full script
+
+```python
+import asyncio
+
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
+from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
+
+
+class KeywordMatchMetric:
+ """Score 1.0 when the agent's answer contains the expected keyword, else 0.0."""
+
+ @property
+ def type(self) -> str:
+ return "keyword_match"
+
+ def output_spec(self) -> list[MetricOutputSpec]:
+ return [MetricOutputSpec.continuous_score("score")]
+
+ async def compute_scores(self, input: MetricInput) -> MetricResult:
+ expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
+ answer = (input.candidate.output_text or "").lower()
+ return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
+
+
+def make_tasks() -> list[AgentEvalTask]:
+ return [
+ AgentEvalTask(
+ id="capital-france",
+ intent="Answer the geography question.",
+ inputs={"instruction": "What is the capital of France?"},
+ reference={"expected": "Paris"},
+ metrics=[KeywordMatchMetric()],
+ ),
+ AgentEvalTask(
+ id="capital-japan",
+ intent="Answer the geography question.",
+ inputs={"instruction": "What is the capital of Japan?"},
+ reference={"expected": "Tokyo"},
+ metrics=[KeywordMatchMetric()],
+ ),
+ ]
+
+
+async def my_agent(task: AgentEvalTask) -> str:
+ # Stand-in agent — replace with a real model or deployed agent later.
+ answers = {
+ "capital-france": "The capital of France is Paris.",
+ "capital-japan": "The capital of Japan is Tokyo.",
+ }
+ return answers[task.id]
+
+
+async def main() -> None:
+ result = await AgentEvaluator().run(
+ tasks=make_tasks(),
+ target=CallableAgentTaskRunner(my_agent),
+ config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
+ )
+ for aggregate in result.summary.scores.scores:
+ print(f"{aggregate.name}: {aggregate.mean}")
+
+
+asyncio.run(main())
+```
+
+Run it:
+
+```bash
+python quickstart.py
+```
+
+Expected output (both answers contain the expected keyword, so the mean is `1.0`):
+
+```
+keyword_match.score: 1.0
+```
+
+Open `./agent-eval-run/report.html` to see the run in a browser.
+
+## Next steps
+
+
+
+
+
+
+- Swap the stand-in agent for a **real model or a deployed agent over HTTP**.
+- Add a **trajectory** metric that scores *how* the agent worked (its tool use), not just the answer.
+- Give a task **more than one metric**, or combine metric outputs into a named **view**.
diff --git a/docs/evaluator/agent-eval/reading-results.mdx b/docs/evaluator/agent-eval/reading-results.mdx
new file mode 100644
index 0000000000..1df54b1fdf
--- /dev/null
+++ b/docs/evaluator/agent-eval/reading-results.mdx
@@ -0,0 +1,94 @@
+---
+title: "Reading Results"
+description: "Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle it writes when you set output_dir, including the browsable HTML report."
+---
+
+`AgentEvaluator().run(...)` returns an `AgentEvalResult`. Set an `output_dir` and it *also* writes a
+**run bundle** to disk. The object and the bundle hold the same data — use the object for programmatic
+follow-up, and the bundle (especially `report.html`) to inspect or share a run.
+
+## The result object
+
+```python
+result = await AgentEvaluator().run(tasks=..., target=...)
+```
+
+| Attribute | What it holds |
+|---|---|
+| `result.run_id` | stable identifier for this run (e.g. `agent-eval-20260715…`) |
+| `result.summary` | aggregated scores and coverage — see [below](#the-summary) |
+| `result.scores` | one entry per **(task, trial, metric)** |
+| `result.trials` | one entry per **trial** |
+| `result.tasks` | the tasks that were evaluated |
+| `result.output_dir` / `result.dashboard_path` | where the bundle and `report.html` were written (when `output_dir` was set) |
+
+### The summary
+
+`result.summary` (an `AgentEvalSummary`):
+
+- **`summary.scores.scores`** — the aggregates. Each is named `.