docs: correct observability and optimization guides

[miyoungc]

Summary

• Correct the Observability curl examples with URL globbing disabled, bearer authentication, HTTP failure handling, and configurable Studio URLs.
• Correct the Experiments ClickHouse health check, shared-Evaluation deletion behavior, soft-deletion lookup guidance, and leaderboard retention semantics.
• Update the insight-driven optimization guide with canonical agent commands, persistence-aware verification, accurate model configuration, and the Eval Author command surface.
• Apply the DORI NVIDIA documentation style rules across the three affected guides.

Why

This is a documentation correctness follow-up to #1048 and #1014. Several examples and behavioral descriptions no longer matched the merged implementation, and remote-deployment examples assumed unauthenticated localhost defaults.

Root cause

The guides described intended behavior while the Intake rollup, deletion, and optimizer command implementations continued to evolve. The examples also did not account for curl URL globbing, authenticated deployments, or the Analyst's profile-based local persistence mode.

Impact

Readers can now run the documented commands against authenticated local or remote deployments and verify the correct persistence target. The Experiments and optimization behavior described in the guides matches the current checked-in implementation.

Validation

• make docs-check (215 MDX files parsed cleanly; gated-link validation passed)
• Extracted Bash examples passed bash -n
• git diff --check
• make docs-broken-links reported no issues in the modified pages; four existing errors remain in unrelated authentication and Studio pages

Summary by CodeRabbit

• Documentation
  • Expanded guidance for AI agent optimization, observability, and experiment workflows.
  • Clarified setup requirements, authentication, workspace configuration, storage, retention, and troubleshooting.
  • Added and updated command examples covering evaluations, telemetry, experiments, scheduling, and Studio verification.
  • Documented evaluator results, score visibility, pagination, deletion behavior, and supported telemetry formats.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-1070-codex-fix-observability-experiment-docs.docs.buildwithfern.com/nemo-platform

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 80e23d8d-daff-4db3-87ab-9e810950fc3c

📥 Commits

Reviewing files that changed from the base of the PR and between 86a3471 and 560bc54.

📒 Files selected for processing (2)

• docs/agents/insight-driven-optimization.mdx
• docs/agents/observability.mdx

🚧 Files skipped from review as they are similar to previous changes (2)

• docs/agents/observability.mdx
• docs/agents/insight-driven-optimization.mdx

---

📝 Walkthrough

Walkthrough

Changes

Documentation workflows

Layer / File(s)	Summary
Insight workflow foundations docs/agents/insight-driven-optimization.mdx	Documents Insight persistence, agent prerequisites, optimization stages, data isolation, and Eval Author commands.
Agent setup and execution docs/agents/insight-driven-optimization.mdx	Updates nemo agents setup, authentication, Harbor evaluation, score uploads, experiment outputs, schedules, flags, and troubleshooting.
Observability ingestion and verification docs/agents/observability.mdx	Adds authenticated telemetry examples, evaluator-result workflows, query guidance, security wording, and retention semantics.
Experiment concepts and setup docs/evaluator/experiments.mdx	Clarifies Experiment membership, evaluator fields, local setup, health checks, creation, durability, and leaderboard verification.
Experiment API operations and retention docs/evaluator/experiments.mdx	Updates authenticated API examples and documents membership, updates, filters, pinning, Pareto settings, deletion, auditing, producers, and rollups.

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#975: Documents workflows enabled by related plugin and feature-flag changes.
• NVIDIA-NeMo/nemo-platform#1048: Introduced content in the observability and Experiments guides.
• NVIDIA-NeMo/nemo-platform#1052: Implements the nemo agents namespace changes reflected in this documentation.

Suggested labels: fix

Suggested reviewers: briannewsom, arpitsardhana

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title concisely describes the documentation corrections in two of the three updated guides and remains related to the primary changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-observability-experiment-docs

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

docs/agents/insight-driven-optimization.mdx (1)

160-179: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Document the lifecycle of Insight evidence.

trace_refs remain persisted after the 90-day TTL removes their spans and trace-index rows. The evidence table omits unavailable traces and reports how many could not be loaded. Update both guides to describe this behavior for platform Insights and local YAML files.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/agents/insight-driven-optimization.mdx` around lines 160 - 179, Update
the Insight evidence lifecycle documentation in
docs/agents/insight-driven-optimization.mdx (lines 160-179) and
docs/agents/observability.mdx (lines 362-363) to state that trace_refs remain
persisted after the 90-day TTL removes their spans and trace-index rows, while
unavailable traces are omitted from the evidence table and the number of omitted
traces is reported; describe this behavior for both platform Insights and local
YAML files.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/agents/insight-driven-optimization.mdx`:
- Around line 223-235: Update the dataset descriptions in the Experimenter
workflow around the validation and training evaluation steps (Lines 126–128) so
they reflect that survivors are first ranked by validation rewards, then
evaluated on the training split. Ensure the documented ordering matches the
implementation and does not describe repeated selection based on held-out
validation scores.
- Around line 428-434: Update the Experimenter evaluation-results description in
docs/agents/insight-driven-optimization.mdx (428-434) to state that each trial
is retrieved by trace_id and every evaluator result is attached to
trace.root_span_id with session_id=trace.session_id, replacing the generic
“attached to the trace” wording. In docs/agents/observability.mdx (254-272),
distinguish this explicit API attachment path from the ATIF-synthesized
harbor.verifier span, with no other behavioral changes.

In `@docs/agents/observability.mdx`:
- Around line 98-108: Update docs/agents/observability.mdx lines 98-108 to use
NMP_BASE_URL=http://localhost:8080 and add the required existing-instance and
/health/ready preflight before CLI/API operations; update lines 124-126 to start
services only when no ready instance exists. In
docs/agents/insight-driven-optimization.mdx lines 299-336, verify /health/ready
before running token and agent commands, preserving the documented
authentication behavior.

In `@docs/evaluator/experiments.mdx`:
- Line 87: Update docs/evaluator/experiments.mdx at lines 87-87 to define each
session without a test_case_id as its own test case and include it in
test_case_count and rollups; at lines 249-252, describe test_case_id as grouping
repeated sessions into one test case without requiring it for rollups; and at
lines 504-504, remove missing test_case_id from the causes of zero metrics.
- Around line 126-131: Update the local setup sections around the environment
exports and CLI examples to use NMP_BASE_URL=http://localhost:8080, check
whether an existing instance is available, and direct users to SETUP.md when it
is not ready. Add a /health/ready verification before running nemo auth token or
other API commands, while keeping the deployed endpoint configurable in a
separate setup branch and applying the same sequence to the referenced sections.
- Around line 126-136: Update the authentication examples around NMP_BASE_URL
and NMP_ACCESS_TOKEN to require a trusted https:// base URL whenever the bearer
token is used. Keep http:// examples only for deployments with authentication
disabled, where the Authorization header and NMP_ACCESS_TOKEN are omitted.

---

Outside diff comments:
In `@docs/agents/insight-driven-optimization.mdx`:
- Around line 160-179: Update the Insight evidence lifecycle documentation in
docs/agents/insight-driven-optimization.mdx (lines 160-179) and
docs/agents/observability.mdx (lines 362-363) to state that trace_refs remain
persisted after the 90-day TTL removes their spans and trace-index rows, while
unavailable traces are omitted from the evidence table and the number of omitted
traces is reported; describe this behavior for both platform Insights and local
YAML files.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ced3c009-b19a-4d96-b457-3942b1997282

📥 Commits

Reviewing files that changed from the base of the PR and between 997172c and 86a3471.

📒 Files selected for processing (3)

• docs/agents/insight-driven-optimization.mdx
• docs/agents/observability.mdx
• docs/evaluator/experiments.mdx

[miyoungc]

CLI and API example verification

Pushed commit 560bc545a842a50b560b132eccbe3e460b53f2a3 with the following code-verified corrections:

1. Use isolated uv dependency groups. Changed uv sync --group insights and uv sync --group experimentalist to uv sync --only-group .... The repository defines enabled-plugins as a default group in pyproject.toml, so --group adds the requested group to all default plugins instead of installing only one side of the optimization loop.
2. Make the Experimenter command shell-safe. Replaced the unquoted angle-bracket placeholders for --insight, --agent, and --config with shell-safe example values. In Bash, the original angle brackets were parsed as redirections.
3. Remove the nonexistent --mode option. The nemo agents experimentalist run Typer signature in plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/cli.py exposes agent, insight, dataset, template, output, workspace, base URL, config, and framework-skill options, but no --mode option.
4. Use the API enum value for span errors. Changed both filter[status]=ERROR examples to filter[status]=error. The status enum in services/intake/src/nmp/intake/spans/domain.py and the generated OpenAPI schema uses lowercase success, error, cancelled, and unknown.
5. Correct the mid-tier model role. Updated the NEMO_EXPERIMENTALIST_MODELS_MID description to trajectory scoring and architecture documentation. The primary Coder uses the smart model; the mid-tier model is used by the architecture-document and trajectory-scoring paths.

I also rechecked the remaining command surfaces against their Typer registrations and implementation code:

• nemo agents analyst run and nemo agents analyst doctor
• nemo agents experimentalist run and nemo agents experimentalist doctor
• nemo agents eval-author --help; the documented placeholder verbs intentionally exit nonzero
• nemo insights analysis enable|status|disable
• nemo services run --services --host --port
• nemo auth token
• Experimenter options --insight, --agent, --train-dataset, --validation-dataset, --task-template, --config, --experiment-dir, --workspace, and --base-url
• Intake and Experiments routes, bearer-token headers, curl -g bracket handling, and the OTLP endpoint used by the examples

Validation results:

• make docs-check: passed with zero Fern errors; all 215 MDX files parsed cleanly; notebook and gated-link validation passed
• Extracted Bash fences from both updated guides piped to bash -n: passed
• Targeted scan for the removed stale forms: passed
• git diff --check: passed
• DCO: commit includes Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

Verification was source-based because this checkout does not contain a synced .venv/bin/nemo. Command registration, option names, routes, schemas, and model use were traced directly in the codebase; shell syntax and documentation structure were validated locally.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	30258/38273	79.1%	63.7%
Integration Tests	17899/36942	48.4%	20.9%