docs(evaluator): add agent (task-driven) evaluation guide

docs/evaluator/agent-eval/evaluate-deployed-agent.mdx

@@ -0,0 +1,210 @@
+---
+title: "Evaluate a Deployed Agent over HTTP"
+description: "Point NeMo Evaluator at an agent reachable over HTTP by describing it as a GenericAgent target — the request to send and where the answer is in the response — then score it exactly like the quickstart."
+---
+
+The [quickstart](/documentation/evaluate-models/agent-eval/quickstart) evaluated an in-process agent.
+This guide evaluates an agent that runs **behind an HTTP endpoint** — a service you (or someone else)
+deployed. Everything else is the same: the same tasks, the same metrics, the same `run()`. Only the
+**target** changes.
+
+You describe an HTTP agent as a `GenericAgent`: its URL, the JSON request to send, and where the
+answer lives in the response. The evaluator then calls that endpoint over real HTTP for each task.
+
+<Note>
+
+This guide assumes you've done the [quickstart](/documentation/evaluate-models/agent-eval/quickstart)
+(install, tasks, metrics, `run()`). It reuses that quickstart's `KeywordMatchMetric` and tasks.
+
+</Note>
+
+## 1. Describe the agent
+
+A `GenericAgent` has three parts you configure:
+
+- **`url`** — where the agent is reachable.
+- **`body`** — the JSON request to POST, as a Jinja template rendered against the task's `inputs`.
+  Reference them directly: the task's instruction is `{{ instruction }}`. The payload is entirely
+  yours — the evaluator sends exactly what `body` produces.
+- **`response_path`** — a JSONPath into the response that selects the agent's answer.
+
+```python
+from nemo_evaluator_sdk.enums import AgentFormat
+from nemo_evaluator_sdk.values import GenericAgent
+
+agent = GenericAgent(
+    name="my-http-agent",
+    url="http://127.0.0.1:8000/agent",
+    format=AgentFormat.GENERIC,
+    body={"message": "{{ instruction }}"},  # -> {"message": "What is the capital of France?"}
+    response_path="$.answer",               # reads {"answer": "..."} from the response
+    # api_key_secret="MY_AGENT_TOKEN",      # optional: env var (local run) or platform secret (job), sent as a bearer token
+)
+```
+
+Set `body` and `response_path` to match **your** agent's request and response shape. Uncomment
+`api_key_secret` for an agent that needs a token — for a local `run()` it names an environment
+variable in your process, and for a submitted job it names a platform secret in the workspace; either
+way its value is sent as a bearer token on each request. The local stand-in below needs no auth, so
+this walkthrough leaves it off.
+
+## 2. Run it against a local stand-in
+
+To try this end to end without deploying anything, stand up a tiny local HTTP server that plays the
+role of the agent. In practice you'd skip this and point `url` at your real endpoint.
+
+```python
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+
+class _AgentHandler(BaseHTTPRequestHandler):
+    def do_POST(self) -> None:
+        payload = json.loads(self.rfile.read(int(self.headers.get("Content-Length", "0"))) or b"{}")
+        text = str(payload.get("message", "")).lower()
+        answer = "Paris" if "france" in text else "Tokyo" if "japan" in text else "unsure"
+        body = json.dumps({"answer": answer}).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args: object) -> None:
+        return  # keep the demo output quiet
+```
+
+## 3. Run the evaluation
+
+With an agent target and no inference function supplied, `run()` generates answers by making real HTTP
+calls to the agent's `url`.
+
+```python
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
+
+result = await AgentEvaluator().run(
+    tasks=make_tasks(),                 # from the quickstart
+    target=agent,
+    config=AgentEvalRunConfig(parallelism=2),
+)
+
+for aggregate in result.summary.scores.scores:
+    print(f"{aggregate.name}: {aggregate.mean}")
+```
+
+## Point at your real agent
+
+To evaluate your own deployed agent, change three things and drop the local server:
+
+1. **`url`** → your agent's endpoint.
+2. **`body`** → the request your agent expects (reference the task's inputs, e.g. `{{ instruction }}`).
+3. **`response_path`** → the JSONPath to the answer in your agent's response.
+
+Add `api_key_secret="MY_AGENT_TOKEN"` if it needs auth (an env var for a local run, a platform secret for a job). If your agent is a **NeMo Agent Toolkit**
+workflow, use `NemoAgentToolkitAgent` instead of `GenericAgent` — it targets a NAT endpoint's fixed
+protocol, so you don't hand-write a `body` template.
+
+## Full script
+
+```python
+import asyncio
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
+from nemo_evaluator_sdk.enums import AgentFormat
+from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
+from nemo_evaluator_sdk.values import GenericAgent
+
+
+class KeywordMatchMetric:
+    @property
+    def type(self) -> str:
+        return "keyword_match"
+
+    def output_spec(self) -> list[MetricOutputSpec]:
+        return [MetricOutputSpec.continuous_score("score")]
+
+    async def compute_scores(self, input: MetricInput) -> MetricResult:
+        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
+        answer = (input.candidate.output_text or "").lower()
+        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
+
+
+def make_tasks() -> list[AgentEvalTask]:
+    return [
+        AgentEvalTask(id="capital-france", intent="Answer the geography question.",
+                      inputs={"instruction": "What is the capital of France?"},
+                      reference={"expected": "Paris"}, metrics=[KeywordMatchMetric()]),
+        AgentEvalTask(id="capital-japan", intent="Answer the geography question.",
+                      inputs={"instruction": "What is the capital of Japan?"},
+                      reference={"expected": "Tokyo"}, metrics=[KeywordMatchMetric()]),
+    ]
+
+
+# A stand-in "deployed agent": a tiny local HTTP server. Replace with your real agent's URL.
+class _AgentHandler(BaseHTTPRequestHandler):
+    def do_POST(self) -> None:
+        payload = json.loads(self.rfile.read(int(self.headers.get("Content-Length", "0"))) or b"{}")
+        text = str(payload.get("message", "")).lower()
+        answer = "Paris" if "france" in text else "Tokyo" if "japan" in text else "unsure"
+        body = json.dumps({"answer": answer}).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args: object) -> None:
+        return
+
+
+async def main() -> None:
+    httpd = ThreadingHTTPServer(("127.0.0.1", 0), _AgentHandler)
+    threading.Thread(target=httpd.serve_forever, daemon=True).start()
+    host, port = httpd.server_address
+
+    agent = GenericAgent(
+        name="my-http-agent",
+        url=f"http://{host}:{port}/agent",
+        format=AgentFormat.GENERIC,
+        body={"message": "{{ instruction }}"},
+        response_path="$.answer",
+    )
+    try:
+        result = await AgentEvaluator().run(
+            tasks=make_tasks(),
+            target=agent,
+            config=AgentEvalRunConfig(parallelism=2),
+        )
+    finally:
+        httpd.shutdown()
+
+    for aggregate in result.summary.scores.scores:
+        print(f"{aggregate.name}: {aggregate.mean}")
+    for trial in result.trials:
+        print(f"  {trial.task_id}: {trial.output.output_text!r}")
+
+
+asyncio.run(main())
+```
+
+Expected output — the answers came back over HTTP and both contain the expected keyword:
+
+```
+keyword_match.score: 1.0
+  capital-france: 'Paris'
+  capital-japan: 'Tokyo'
+```
+
+## Next steps
+
+<Cards>
+  <Card title="Agent Evaluation (concepts)" href="/documentation/evaluate-models/agent-eval" />
+  <Card title="Agent Evaluation Quickstart" href="/documentation/evaluate-models/agent-eval/quickstart" />
+</Cards>
+
+- Score the agent's **trajectory** (its tool use), not just the final answer.
+- Give a task **multiple metrics** and combine them into a named **view**.

docs/evaluator/agent-eval/harbor-runner.mdx

@@ -0,0 +1,167 @@
+---
+title: "Evaluate a Harbor Task Suite"
+description: "Run an existing Harbor task dataset through NeMo Evaluator's Harbor runner — Harbor executes each task in a Docker sandbox and its verifier emits a reward, which the SDK scores and reports like any other agent-eval run."
+---
+
+[Harbor](https://www.harborframework.com) is a container-based harness for agentic tasks: it runs each
+task in a Docker sandbox, lets an agent work in it, then runs a **verifier** that emits a **reward**
+(see Harbor's [Core Concepts](https://www.harborframework.com/docs/core-concepts) for its task, trial,
+and job model). If you already have Harbor task datasets, the **Harbor runner** runs them and scores
+the verifier reward through agent-eval — the same
+[`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the same result and bundle as the
+[quickstart](/documentation/evaluate-models/agent-eval/quickstart). Only the runner changes.
+
+<Warning>
+
+Unlike the quickstart, this runner is **not** zero-dependency — it shells out to Harbor and Docker:
+
+- **Python ≥ 3.12**
+- **Docker** installed and running
+- **Harbor**, installed separately: `uv pip install "harbor>=0.16.1"`. Harbor is intentionally **not**
+  a dependency of `nemo-platform[nemo-evaluator-sdk]`, so the rest of the SDK stays lightweight.
+
+The runner raises a clear error pointing at this install step if `harbor` is missing.
+
+</Warning>
+
+## The dataset
+
+A Harbor dataset is a directory of **task folders**. For discovery, the runner needs two files per
+task — the rest of the task format (environment, verifier, and solution config) is Harbor's own:
+
+```
+my-suite/
+  hello-world/
+    task.toml         # [task] name = "harbor/hello-world"
+    instruction.md    # Create a file called hello.txt with "Hello, world!" as the content.
+  ...
+```
+
+`discover_harbor_tasks` reads each folder into an `AgentEvalTask`: the `[task] name` becomes the task
+`id` and its human-readable `intent`, and `instruction.md` becomes `inputs["instruction"]` — the
+instruction the agent is prompted with. See
+[Harbor's task documentation](https://www.harborframework.com/docs/tasks) for the full task format.
+
+<Note>
+
+The repo ships a one-task
+[example dataset](https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/packages/nemo_evaluator_sdk/examples/harbor/hello_world_dataset)
+you can clone and point at (it is not shipped in the installed wheel). Or point the dataset path at
+your own Harbor suite.
+
+</Note>
+
+## Run it
+
+A Harbor run is a normal agent-eval run: `AgentEvaluator().run(tasks=..., target=runner)`, exactly like
+the [quickstart](/documentation/evaluate-models/agent-eval/quickstart) (a callable) and the
+[deployed-agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) guide (an HTTP
+target). Here the target is a `HarborAgentTaskRunner`.
+
+```python
+import asyncio
+from pathlib import Path
+
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import (
+    HarborAgentTaskRunner,
+    HarborRuntimeConfig,
+    discover_harbor_tasks,
+)
+
+
+async def main() -> None:
+    # INPUT: your Harbor task suite (a directory of task folders).
+    tasks = discover_harbor_tasks("path/to/my-suite")
+
+    runner = HarborAgentTaskRunner(
+        config=HarborRuntimeConfig(
+            # OUTPUT: where Harbor writes its <job_name>/ results tree (not the dataset).
+            jobs_dir=Path("./harbor-jobs"),
+            agent_name="oracle",  # a built-in Harbor agent — see "Choosing an agent"
+        )
+    )
+
+    # Same AgentEvaluator as every agent-eval run; only the target changes.
+    result = await AgentEvaluator().run(tasks=tasks, target=runner)
+
+    for aggregate in result.summary.scores.scores:
+        print(f"{aggregate.name}: {aggregate.mean}")
+
+
+asyncio.run(main())
+```
+
+The three pieces map onto the model: `discover_harbor_tasks` turns the suite into tasks,
+`HarborAgentTaskRunner` runs the Harbor job and returns one trial per Harbor trial, and `AgentEvaluator`
+scores each trial's reward with `HarborRewardMetric`.
+
+<Note>
+
+**`dataset_path` is input; `jobs_dir` is output.** The dataset is your read-only task suite. `jobs_dir`
+is a directory the runner writes into — Harbor's per-trial results land under `jobs_dir/<job_name>/`,
+and that directory doubles as a re-run cache (see [below](#attempts-concurrency-and-caching)).
+
+</Note>
+
+The `oracle` agent is Harbor's reference agent — it produces a passing trial on a well-formed task —
+so for the one-task example above the verifier reward is `1.0`:
+
+```
+harbor_reward.reward: 1.0
+```
+
+Swap `agent_name` (or `agent_import_path`) for your own agent to get a real score.
+
+## Choosing an agent
+
+`HarborRuntimeConfig` decides what runs inside each sandbox (see
+[Harbor's agent documentation](https://www.harborframework.com/docs/agents)):
+
+- **`agent_name`** — a built-in Harbor agent (for example `"oracle"`, Harbor's reference agent, handy
+  as a smoke test that the harness and dataset are wired up).
+- **`agent_import_path`** — your own Harbor agent, e.g. `"my_agent_module:MyAgent"`. Set `agent_dir`
+  as well when it's a loose file rather than an installed package. Overrides `agent_name`.
+- **`agent_model_name`** — the model slug handed to the agent.
+
+## How scoring works
+
+1. Harbor runs each task in its sandbox and writes a `<task>__<hash>/result.json` per trial, including
+   the reward from the task's [verifier](https://www.harborframework.com/docs/tasks).
+2. The runner reads that reward onto each trial's metadata, and `HarborRewardMetric` scores it — the
+   `reward` value, or `0.0` for a trial whose verifier emitted none.
+3. `result.summary` aggregates the reward across tasks (`harbor_reward.reward`), `result.trials` holds
+   each trial's status and evidence, and — if you pass a `config` with an `output_dir` to `run()` — the
+   run **bundle** (including `report.html`) is written like any other agent-eval run.
+
+## Attempts, concurrency, and caching
+
+- **`n_attempts`** — trials Harbor runs per task; **`n_concurrent_trials`** — how many run at once.
+- The job directory doubles as a **cache**. Pin a stable `job_name` and a completed job whose results
+  already cover every requested task is re-scored instead of re-run. The default timestamped `job_name`
+  always runs fresh; set `force_rerun=True` to delete an existing job dir first.
+
+## Shortcut: `run_harbor_eval`
+
+When a run is exactly "one Harbor suite, scored by its reward," `run_harbor_eval` collapses the three
+steps above — discover, run, score — into a single call:
+
+```python
+from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
+
+result = await run_harbor_eval(
+    HarborRuntimeConfig(jobs_dir="./harbor-jobs", agent_name="oracle"),
+    dataset_path="path/to/my-suite",
+)
+```
+
+It uses the same `AgentEvaluator` and `HarborRewardMetric` under the hood. Prefer the explicit form
+above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across
+targets.
+
+## Next steps
+
+<Cards>
+  <Card title="Agent Evaluation (concepts)" href="/documentation/evaluate-models/agent-eval" />
+  <Card title="Agent Evaluation Quickstart" href="/documentation/evaluate-models/agent-eval/quickstart" />
+</Cards>