Skip to content

feat(intake): add canonical experiment_id filter to evaluations list [ASE-716] - #949

Merged
shanaiabuggy merged 3 commits into
mainfrom
sbuggy/ase-716-add-canonical-experiment_id-filter-to-evaluations-list
Jul 28, 2026
Merged

feat(intake): add canonical experiment_id filter to evaluations list [ASE-716]#949
shanaiabuggy merged 3 commits into
mainfrom
sbuggy/ase-716-add-canonical-experiment_id-filter-to-evaluations-list

Conversation

@shanaiabuggy

@shanaiabuggy shanaiabuggy commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

The evaluations write path is canonically experiment_ids, but the list filter only exposed the deprecated experiment_group_id, forcing producers to write one key and read back by another. This brings the filter in line.

  • Add a canonical experiment_id filter to EvaluationFilter that routes through the same group-membership match (experiment_ids $contains + the legacy scalar) as the existing rewrite.
  • Mark the experiment_group_id filter param deprecated=True, kept as a fully-working alias (non-breaking) — mirrors how the request body deprecates its experiment_group_id field.
  • _rewrite_group_filter now recognizes both keys; updated the list-endpoint filter description.
  • Regenerated the OpenAPI spec.

Testing

  • New test_evaluation_filter_experiment_id_matches_deprecated_alias: membership set via the canonical experiment_ids write field, then asserts filter[experiment_id] and filter[experiment_group_id] return the same set.
  • ruff clean; intake CRUD integration suite: 38 passed.

Follow-up

Removal of the deprecated experiment_group_id filter param is tracked as a breaking change in ASE-615.

Resolves ASE-716.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added canonical experiment_id filtering for evaluations by experiment-group membership.
    • Kept experiment_group_id as a deprecated compatibility alias.
  • Documentation
    • Updated evaluation listing API docs and the EvaluationFilter schema to reflect the new canonical filter and aliasing.
  • Bug Fixes
    • Standardized server and UI filtering so both filter names resolve to equivalent results.
  • Tests
    • Added integration coverage confirming experiment_id matches the deprecated alias behavior.

…[ASE-716]

The evaluations write path is canonically `experiment_ids`; bring the list filter in line so
producers stop having to write `experiment_ids` and read back by the deprecated key.

- Add a canonical `experiment_id` filter to EvaluationFilter that routes through the same
  group-membership match (experiment_ids $contains + legacy scalar) as before.
- Mark the `experiment_group_id` filter param deprecated=True, kept as a working alias.
- _rewrite_group_filter now recognizes both keys; update the list-endpoint filter description.
- Add a test: filtering by experiment_id returns the same set as experiment_group_id, both
  resolving membership over experiment_ids.
- Regenerate the OpenAPI spec.

Removal of the deprecated experiment_group_id filter param is tracked in ASE-615.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
@shanaiabuggy
shanaiabuggy requested review from a team as code owners July 28, 2026 18:06
@github-actions github-actions Bot added the feat label Jul 28, 2026
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a2bfe69d-6e28-4c12-a5e5-70b9cbfd08aa

📥 Commits

Reviewing files that changed from the base of the PR and between 132cb2c and 1971b4c.

⛔ Files ignored due to path filters (5)
  • sdk/python/nemo-platform/.nmpcontext/openapi.yaml is excluded by !sdk/**
  • sdk/python/nemo-platform/src/nemo_platform/resources/evaluations/evaluations.py is excluded by !sdk/**
  • sdk/python/nemo-platform/src/nemo_platform/types/evaluations/evaluation_filter_param.py is excluded by !sdk/**
  • sdk/python/nemo-platform/src/nemo_platform/types/evaluations/evaluation_list_params.py is excluded by !sdk/**
  • sdk/python/nemo-platform/tests/api_resources/test_evaluations.py is excluded by !sdk/**
📒 Files selected for processing (5)
  • web/packages/studio/src/components/ExperimentGroupEditModal/index.tsx
  • web/packages/studio/src/components/charts/ExperimentGroupParetoChart/useParetoEvaluations.ts
  • web/packages/studio/src/components/dataViews/ExperimentGroupDataView/useExperimentGroupEvaluations.test.ts
  • web/packages/studio/src/components/dataViews/ExperimentGroupDataView/useExperimentGroupEvaluations.ts
  • web/packages/studio/src/routes/EvaluationSessionDetailRoute/useSessionCompareRuns.ts

📝 Walkthrough

Walkthrough

The evaluations filter contract now defines experiment_id as canonical, retains experiment_group_id as deprecated, rewrites both fields through group-membership matching, and updates Studio queries and integration coverage.

Changes

Evaluation filter aliasing

Layer / File(s) Summary
Canonical filter contract
services/intake/src/nmp/intake/api/v2/experiments/schemas.py, openapi/.../openapi.yaml, openapi/ga/individual/platform.openapi.yaml, services/intake/src/nmp/intake/api/v2/experiments/endpoints.py
experiment_id is documented as the canonical membership filter, while experiment_group_id is marked as its deprecated alias.
Group filter rewriting
services/intake/src/nmp/intake/api/v2/experiments/endpoints.py
Equality filters for both identifier fields use shared experiment-group membership matching logic.
Alias compatibility validation
services/intake/tests/integration/test_experiments_crud.py
Integration coverage confirms both filter parameters return the same evaluations for a group.
Studio filter migration
web/packages/studio/src/components/ExperimentGroupEditModal/index.tsx, web/packages/studio/src/components/charts/ExperimentGroupParetoChart/useParetoEvaluations.ts, web/packages/studio/src/components/dataViews/ExperimentGroupDataView/*, web/packages/studio/src/routes/EvaluationSessionDetailRoute/useSessionCompareRuns.ts
Studio evaluation queries and cache invalidation now use experiment_id for experiment-group scoping, with updated query-key expectations.

Possibly related PRs

Suggested reviewers: briannewsom

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding the canonical experiment_id filter to evaluations list.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sbuggy/ase-716-add-canonical-experiment_id-filter-to-evaluations-list

Comment @coderabbitai help to get the list of available commands.

…ilter

Ran `make stainless` after the EvaluationFilter change — the generated Python SDK now exposes
the canonical `experiment_id` filter and marks `experiment_group_id` as a deprecated alias.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 27559/35314 78.0% 62.5%
Integration Tests 16094/34032 47.3% 19.8%

Migrate the FE evaluations-list filter call sites off the deprecated experiment_group_id to
the canonical experiment_id (leaderboard, Pareto chart, edit modal, session compare). The
pin/unpin invalidation query-key moves in lockstep so its partial-key match keeps working.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
@shanaiabuggy
shanaiabuggy enabled auto-merge July 28, 2026 21:11
@shanaiabuggy
shanaiabuggy added this pull request to the merge queue Jul 28, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 28, 2026
@shanaiabuggy
shanaiabuggy added this pull request to the merge queue Jul 28, 2026
Merged via the queue into main with commit 3886181 Jul 28, 2026
60 checks passed
@shanaiabuggy
shanaiabuggy deleted the sbuggy/ase-716-add-canonical-experiment_id-filter-to-evaluations-list branch July 28, 2026 23:19
ngoncharenko pushed a commit that referenced this pull request Jul 29, 2026
…[ASE-716] (#949)

* feat(intake): add canonical experiment_id filter to evaluations list [ASE-716]

The evaluations write path is canonically `experiment_ids`; bring the list filter in line so
producers stop having to write `experiment_ids` and read back by the deprecated key.

- Add a canonical `experiment_id` filter to EvaluationFilter that routes through the same
  group-membership match (experiment_ids $contains + legacy scalar) as before.
- Mark the `experiment_group_id` filter param deprecated=True, kept as a working alias.
- _rewrite_group_filter now recognizes both keys; update the list-endpoint filter description.
- Add a test: filtering by experiment_id returns the same set as experiment_group_id, both
  resolving membership over experiment_ids.
- Regenerate the OpenAPI spec.

Removal of the deprecated experiment_group_id filter param is tracked in ASE-615.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>

* chore(sdk): regenerate Python SDK for the experiment_id evaluations filter

Ran `make stainless` after the EvaluationFilter change — the generated Python SDK now exposes
the canonical `experiment_id` filter and marks `experiment_group_id` as a deprecated alias.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>

* refactor(studio): use the canonical experiment_id evaluations filter

Migrate the FE evaluations-list filter call sites off the deprecated experiment_group_id to
the canonical experiment_id (leaderboard, Pareto chart, edit modal, session compare). The
pin/unpin invalidation query-key moves in lockstep so its partial-key match keeps working.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>

---------

Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants