Skip to content

feat(measurement): add W&B benchmark identity config - #236

Merged
lipikaramaswamy merged 5 commits into
mainfrom
lipikaramaswamy/feature/wandb-benchmark-identity
Jul 31, 2026
Merged

feat(measurement): add W&B benchmark identity config#236
lipikaramaswamy merged 5 commits into
mainfrom
lipikaramaswamy/feature/wandb-benchmark-identity

Conversation

@lipikaramaswamy

Copy link
Copy Markdown
Collaborator

Summary

  • Add optional benchmark identity metadata to W&B strict import config and run config.
  • Add import_wandb_run.py flags for benchmark role/kind, suite version, branch, PR/release metadata, Anonymizer commit, config ID, and mode.
  • Validate that kind=pr and kind=branch include a PR number so candidate benchmark runs can be filtered consistently in W&B.
  • Document the benchmark identity fields used by the experiment runner.

Tests

  • python3 -m py_compile tools/measurement/import_wandb_run.py tools/measurement/measurement_tools/wandb_models.py
  • uv run pytest tests/tools/test_measurement_strict_import_publisher.py tests/tools/test_measurement_wandb_logging.py
  • uv run ruff check tools/measurement/import_wandb_run.py tools/measurement/measurement_tools/wandb_models.py tests/tools/test_measurement_strict_import_publisher.py tests/tools/test_measurement_wandb_logging.py
  • git diff --check

Notes

The companion anonymizer-experiments branch wires these fields through suite requests and W&B publication.

Signed-off-by: lipikaramaswamy <lramaswamy@nvidia.com>
@lipikaramaswamy
lipikaramaswamy requested review from a team as code owners July 31, 2026 05:55
@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds validated benchmark identity metadata to strict W&B imports.

  • Adds CLI flags and typed W&B config fields for benchmark role, kind, suite, branch, commit, PR/release, configuration, and mode.
  • Enforces compatible role/kind pairs and consistent full/short commit identifiers.
  • Refreshes benchmark identity fields when an already-complete import is resumed.
  • Preserves launcher-provided WANDB_API_KEY during W&B SDK execution and updates documentation and tests accordingly.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
tools/measurement/import_wandb_run.py Builds validated optional benchmark identity metadata from CLI arguments and includes it in sealed-import payloads.
tools/measurement/measurement_tools/wandb_models.py Defines and validates benchmark identity metadata and projects it into W&B configuration.
tools/measurement/measurement_tools/wandb_setup.py Refreshes identity fields on completed resumed imports and allows an explicitly inherited W&B API key.
tests/tools/test_measurement_wandb_logging.py Covers commit consistency, role/kind compatibility, PR requirements, and API-key preservation.
tests/tools/test_measurement_strict_import_publisher.py Covers identity projection and identity-only refreshes for completed imports.
tools/measurement/README.md Documents launcher authentication and the new benchmark identity flags.

Reviews (7): Last reviewed commit: "fix(measurement): refresh W&B identity o..." | Re-trigger Greptile

Comment thread tools/measurement/import_wandb_run.py
Comment thread tools/measurement/measurement_tools/wandb_models.py
Signed-off-by: lipikaramaswamy <lramaswamy@nvidia.com>
Signed-off-by: lipikaramaswamy <lramaswamy@nvidia.com>
Signed-off-by: lipikaramaswamy <lramaswamy@nvidia.com>
Comment thread tools/measurement/import_wandb_run.py
@lipikaramaswamy
lipikaramaswamy force-pushed the lipikaramaswamy/feature/wandb-benchmark-identity branch from 545d20e to 6de7bfe Compare July 31, 2026 19:52
Signed-off-by: lipikaramaswamy <lramaswamy@nvidia.com>
@lipikaramaswamy
lipikaramaswamy force-pushed the lipikaramaswamy/feature/wandb-benchmark-identity branch from 6de7bfe to a85b2fc Compare July 31, 2026 20:02
@lipikaramaswamy
lipikaramaswamy merged commit 38c23e0 into main Jul 31, 2026
12 checks passed
@lipikaramaswamy
lipikaramaswamy deleted the lipikaramaswamy/feature/wandb-benchmark-identity branch July 31, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants