feat(eval-author): narrate insight-mode progress via Experimentalist RunReporter - #1047
Merged
aleckhoury merged 3 commits intoAug 3, 2026
Merged
Conversation
…ist RunReporter Signed-off-by: Alec Khoury <akhoury@nvidia.com>
…RunReporter Reuse the parent RunReporter so Experimentalist insight runs show Eval Author start, phase, and complete lines instead of going quiet mid-pipeline. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Contributor
Contributor
📝 WalkthroughWalkthroughChangesEval Author reporting
Sequence Diagram(s)sequenceDiagram
participant ExperimentalistLoop
participant EvalAuthor
participant RunReporter
ExperimentalistLoop->>EvalAuthor: construct with reporter
EvalAuthor->>RunReporter: report startup and phase progress
EvalAuthor->>RunReporter: report completion after finalization
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py (1)
371-390: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winCount completed trace analyses, not successful diagnostics.
Line 387 uses
len(diagnostics). Failed analyses are excluded. If every analysis fails, output remainstrace 0/≤Neven after all analyses finish.Use
len(raw_diagnostics)or a processed-result counter. Keep failures innote(...). Add a regression test with a failed analysis that assertstrace N/≤N.Proposed fix
phase="eval author · analyzing traces", - completed=len(diagnostics), + completed=len(raw_diagnostics), total=len(trials),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py` around lines 371 - 390, Update the trace-analysis progress call in the surrounding analysis loop to report processed analyses rather than successful entries in diagnostics: use raw_diagnostics length or an equivalent counter that includes failures. Preserve the existing failure logging and reporter.note behavior, and add a regression test covering a failed analysis that verifies progress reaches trace N/≤N.
🧹 Nitpick comments (2)
docs/superpowers/specs/2026-08-03-ase-749-eval-author-progress-design.md (1)
1-9: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd required document navigation.
Classify this page as an EXPLANATION. Add prerequisites near the top. Add a
Next Stepssection with cross-links at the end.As per coding guidelines, “Always list prerequisites at the top of documentation pages” and “Include 'Next Steps' section at the end with cross-links to related documentation content.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/superpowers/specs/2026-08-03-ase-749-eval-author-progress-design.md` around lines 1 - 9, Classify the document as an EXPLANATION, add a prerequisites section near the top using the repository’s standard documentation format, and append a Next Steps section at the end with cross-links to related documentation. Preserve the existing design content and ensure both navigation sections follow the project’s documentation guidelines.Source: Coding guidelines
plugins/nemo-eval-author/tests/test_eval_author_agent.py (1)
93-102: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
RunReporteras the concrete test-helper type. Import it at module scope. AnnotatereporterasRunReporter | Noneand_string_reporter()astuple[RunReporter, io.StringIO].🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py` around lines 93 - 102, Import RunReporter at module scope, annotate the reporter parameter in the EvalAuthor test helper with RunReporter | None, and update _string_reporter() to return tuple[RunReporter, io.StringIO]; apply the corresponding type annotation at the sibling site in plugins/nemo-eval-author/tests/test_eval_author_agent.py lines 693-697.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py`:
- Around line 371-390: Update the trace-analysis progress call in the
surrounding analysis loop to report processed analyses rather than successful
entries in diagnostics: use raw_diagnostics length or an equivalent counter that
includes failures. Preserve the existing failure logging and reporter.note
behavior, and add a regression test covering a failed analysis that verifies
progress reaches trace N/≤N.
---
Nitpick comments:
In `@docs/superpowers/specs/2026-08-03-ase-749-eval-author-progress-design.md`:
- Around line 1-9: Classify the document as an EXPLANATION, add a prerequisites
section near the top using the repository’s standard documentation format, and
append a Next Steps section at the end with cross-links to related
documentation. Preserve the existing design content and ensure both navigation
sections follow the project’s documentation guidelines.
In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py`:
- Around line 93-102: Import RunReporter at module scope, annotate the reporter
parameter in the EvalAuthor test helper with RunReporter | None, and update
_string_reporter() to return tuple[RunReporter, io.StringIO]; apply the
corresponding type annotation at the sibling site in
plugins/nemo-eval-author/tests/test_eval_author_agent.py lines 693-697.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e0e8aaf8-692e-47e9-b832-ee9344cfbba4
📒 Files selected for processing (6)
docs/superpowers/specs/2026-08-03-ase-749-eval-author-progress-design.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.pyplugins/nemo-eval-author/tests/test_eval_author_agent.pyplugins/nemo-eval-author/tests/test_plugin_boundary.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py
Contributor
|
BrianNewsom
approved these changes
Aug 3, 2026
BrianNewsom
left a comment
BrianNewsom
left a comment
Contributor
There was a problem hiding this comment.
Remove spec and check on optionality, then ship it.
Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
deleted the
ASE-749-improve-eval-author-progress-output/akhoury
branch
August 3, 2026 22:15
sklinglernv
added a commit
that referenced
this pull request
Aug 4, 2026
Two commits on main landed in files this branch rewrites: - #1047 (eval-author: narrate insight-mode progress via RunReporter) added `nemo_experimentalist_plugin.experimentalist.reporting` to the Eval Author boundary ratchet, which this branch had split into `_SHARED_LAYER_A` and `_BORROWED_BEHAVIOUR`. Resolved by keeping the split and filing the new import under borrowed behaviour: RunReporter is reused behaviour, not entity contract, so it belongs on the list that may only shrink. - #1053 (experimentalist: honor remote auth discovery) auto-merged. Also updates docs/agents/insight-driven-optimization.mdx, new on main, which documented the pre-rename variables and model defaults this branch removes. Signed-off-by: Severin Klingler <sklingler@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RunReporterin Eval Author when invoked from insight mode, so the parent run keeps narrating instead of going quiet during materialize / analyze / author / repair.EvalAuthor._run; Experimentalistloop.pypasses the parent reporter. Standalonerun_eval_authorstays unchanged (reporter=None).experimentalist.reportingon the Eval Author plugin boundary and cover the new narration with unit tests. Closes ASE-749; mirrors PR #965.Validation screenshot:
Test plan
uv run pytest plugins/nemo-eval-author/tests/test_eval_author_agent.py plugins/nemo-eval-author/tests/test_plugin_boundary.py plugins/nemo-eval-author/tests/test_eval_author_run.py -quv run ruff check/ruff format --checkon touched filesuv run --frozen ty checkon Eval Authoragent.py/run.pynemo agents experimentalist runin insight mode and confirm Eval Author start/phase/complete lines appear on the same stderr stream as Experimentalist progressSummary by CodeRabbit
New Features
Bug Fixes
Documentation