feat(experimentalist): add the airline, retail, and telecom tau3 suites - #980
Merged
Conversation
…itions The coverage test named one manifest, so a second suite's expected_task_count, task-ID prefix, or fast entries went unchecked until a run reached Harbor Hub. It now runs over every file in suites/, substituting the manifest's own quality partition for the Hub's task list. That still catches a count that disagrees with the split, a fast entry outside the domain, overlapping splits, and an unresolvable framework skill, which is how a hand-written manifest goes wrong. The Terminal Bench task-ID correction keeps its own assertion; its split sizes were only restating the coverage check. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Aditya Pandey <aditya@autospace.co>
sierra-research/tau3-bench@1 holds 375 tasks across four tau2 domains; only banking_knowledge was expressed as a suite. The manifest format already scopes a package by task-ID prefix, so the other three need manifests and nothing else: the two tau3 configs and examples/tau3-nooa-agent are domain-independent, because the domain policy arrives at runtime in the task instruction. Canonical IDs come from Harbor's adapters/tau3-bench, which names each task tau3-<domain>-<slugified upstream task ID>. Re-deriving them from that adapter against a tau2-bench checkout reproduces the banking manifest exactly, which is what makes it a usable source for the other three. Airline, retail, and telecom carry an upstream split_tasks.json whose train and test lists partition the domain exactly, so test is taken verbatim and only validation is carved out, every third train entry in upstream order. Banking instead needed a whole split invented, hence its separate provenance. reward_basis decides whether the fast partition can score without the LLM judge. Airline scores on database state and a substring check, telecom on environment assertions and expected actions, so both are judge-free. Retail cannot be: 112 of its 114 tasks carry an NL_ASSERTION, leaving too few pure-database tasks to fill a split, so its fast partition takes the first 6/3/3 like the others and scores through the judge that models.user_simulator already configures. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Aditya Pandey <aditya@autospace.co>
Taking the first 6/3/3 of each quality split made telecom's smoke partition twelve near-identical tasks: its IDs sort by issue type and the first twelve are all mms_issue, so a smoke run exercised one of the domain's flows and would miss a harness break in any other. Each of the three splits is now sampled at a fixed stride instead, which spans the split at the same size and needs no seed to reproduce. Telecom's fast partition now covers three issue types. Banking is untouched: it picks from its pure-database-state subset for a different reason. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Aditya Pandey <aditya@autospace.co>
… claims Review of the three new manifests found the surrounding claims weaker than the data. Four corrections, no task ID changes: Dropping Terminal Bench's split-size assertions lost coverage rather than restating it: the canonical check only pins the total, so rebalancing 38/25/26 to 37/26/26 passed. Every suite's documented quality and fast sizes are now recorded in one table the parametrized test asserts, which also pins the four tau3 splits the README publishes. The test docstring no longer implies the substituted task list catches a wrong task_id_prefix, a dropped ID, or a stale resolved_ref; it cannot. The banking cross-check is narrower than claimed. Banking is the one domain Harbor's adapter enumerates from a task directory instead of split_tasks.json, so reproducing it never exercises the selection path the other three use. The README now says so, and cites the adapter's own per-domain counts as the independent check on sizes. Reward basis is per task, not per domain. Retail does not "always" score through the judge — tasks 33 and 34 are database-state only — and airline is scored more leniently than upstream tau2 airline, because its reward_basis omits the nl_assertions every task carries and only 6 of 50 assert communicate_info. Airline's 20/10/20 split also leaves the optimizer the least signal of any suite, which is worth knowing before reading its numbers. The split rules were not reproducible as written: "every third entry" did not say which third, and the fast stride omitted its formula. Both are now spelled out, and the reward-basis sentences that justified nothing in their own manifest are gone. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Aditya Pandey <aditya@autospace.co>
The offline checks cover different things: the adapter's naming rule accounts for every ID, its recorded metadata only for the per-domain counts. Claiming both agreed on both overstated the weaker one. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Aditya Pandey <aditya@autospace.co>
Contributor
|
callingmedic911
marked this pull request as ready for review
July 29, 2026 22:57
Contributor
📝 WalkthroughWalkthroughThe Experimentalist benchmark now ships airline, retail, and telecom TAU3 suites alongside existing suites. Documentation covers all TAU3 domains, and tests validate every shipped suite’s metadata, framework skills, and partition sizes. ChangesTAU3 Domain Suite Expansion
Sequence Diagram(s)sequenceDiagram
participant ShippedSuiteYAML
participant BenchmarkTest
participant CanonicalSuiteValidator
ShippedSuiteYAML->>BenchmarkTest: load each shipped suite
BenchmarkTest->>CanonicalSuiteValidator: validate suite metadata and partitions
CanonicalSuiteValidator-->>BenchmarkTest: validated suite and framework skill paths
BenchmarkTest->>ShippedSuiteYAML: compare quality and fast split sizes
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/benchmarks/README.md`:
- Around line 68-70: Correct the README statement describing Hub validation:
update it to say that run_benchmark validates the canonical suite during normal
runs as well as with --validate-only, while --validate-only only skips Docker
and model execution. Preserve the surrounding explanation about the banking
adapter and published package.
In `@plugins/nemo-experimentalist/benchmarks/suites/tau3-airline.yaml`:
- Line 12: Update the quality and fast task IDs in
plugins/nemo-experimentalist/benchmarks/suites/tau3-airline.yaml at lines 12-12
to use the tau3-bench__tau3-airline- prefix; apply the equivalent canonical
prefix tau3-bench__tau3-retail- to all quality and fast IDs in
plugins/nemo-experimentalist/benchmarks/suites/tau3-retail.yaml at lines 12-12,
and tau3-bench__tau3-telecom- in
plugins/nemo-experimentalist/benchmarks/suites/tau3-telecom.yaml at lines 12-12.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 328d5a34-3ce6-4ecd-8870-7ce97c7af593
📒 Files selected for processing (5)
plugins/nemo-experimentalist/benchmarks/README.mdplugins/nemo-experimentalist/benchmarks/suites/tau3-airline.yamlplugins/nemo-experimentalist/benchmarks/suites/tau3-retail.yamlplugins/nemo-experimentalist/benchmarks/suites/tau3-telecom.yamlplugins/nemo-experimentalist/tests/test_experimentalist_benchmark.py
nicot
approved these changes
Jul 29, 2026
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #935, which added
banking_knowledge.sierra-research/tau3-bench@1holds 375 tasks across four tau2 domains; this expresses the other three as suites.Summary by CodeRabbit
New Features
--suiteoption.Documentation