Skip to content

ci: pin wheel install-test deps to uv.lock (supply-chain) - #649

Merged
SandyChapman merged 1 commit into
mainfrom
ci-pin-wheel-smoke-test-deps/schapman
Jul 13, 2026
Merged

ci: pin wheel install-test deps to uv.lock (supply-chain)#649
SandyChapman merged 1 commit into
mainfrom
ci-pin-wheel-smoke-test-deps/schapman

Conversation

@SandyChapman

@SandyChapman SandyChapman commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Problem

The wheel install-smoke-test jobs (nemo-platform wheel build + test, nemo-platform-plugin wheel build + test) install the freshly-built wheel with no dependency pinning:

uv tool install --force --python "${PYTHON_VERSION}" "${WHEEL}[services]"   # nemo-platform
uv pip install  --python .venv/bin/python "${WHEEL}"                        # plugin

Neither uses --constraints, uv.lock, --frozen, or hashes, so they resolve the entire transitive closure fresh from PyPI, latest-in-range. That's:

  • A supply-chain risk — these jobs then execute the installed code (CLI smoke test / imports) in the runner, so a compromised newer in-range release would run with the runner's permissions/secrets.
  • Non-reproducible — a run's dependency set drifts as PyPI publishes new releases.

This is what let litellm==1.92.0 slip in over the locked 1.90.2 and turn the py3.14 job red (1.92.0 added a Rust/PyO3 core that fails to build on Python 3.14). (see this CI link)

Note: the rest of CI (unit/integration tests, builds) already runs --frozen against uv.lock; this gap was specific to these two install-test steps.

Fix

Export the locked dependency set from uv.lock and pass it as --constraints to both installs:

uv export --frozen --all-packages --no-emit-workspace --no-emit-local --no-hashes \
  -o "${RUNNER_TEMP}/wheel-constraints.txt"
uv tool install --force --python "${PYTHON_VERSION}" \
  --constraints "${RUNNER_TEMP}/wheel-constraints.txt" "${WHEEL}[services]"
  • --all-packages pins the full external dependency universe of the workspace.
  • --no-emit-workspace --no-emit-local drop the local/workspace path & editable entries (which aren't installable as constraints), leaving only PyPI-pinned versions.
  • Now the smoke test installs only versions vetted in uv.lock; a newer (potentially malicious) in-range release can no longer be pulled.

Side effect: this also fixes the py3.14 wheel+test failure — the constraints force litellm==1.90.2 (pure-Python wheel, installs cleanly on 3.14).

Verification

  • uv pip compile with the exported constraints resolves litellm==1.90.2; without it, 1.92.0.
  • The exported constraints file has 0 local/editable/path entries (clean, install-safe).
  • [services] extra has no heavy/native deps (no torch/vllm/flashinfer), so pinning to the lock is safe across the Python matrix.
  • Workflow YAML validated; Run actionlint in CI covers the shell.

Follow-up (not in this PR)

Hash-pinning (--require-hashes) is stronger defense-in-depth but needs the local wheel hashed too; version-pinning here closes the practical "auto-pull a newer release" vector first.

🤖 Generated with Claude Code

Summary by CodeRabbit

Summary by CodeRabbit

  • Chores
    • Improved CI wheel smoke tests by installing dependencies from committed, hash-pinned lock/constraint files.
    • Updated the wheel install/validation flow to install the wheels without resolving new dependencies.
    • Added lockfile artifacts for the platform services and plugin wheels, plus a helper script to regenerate them for consistent, reproducible builds.

@SandyChapman
SandyChapman requested a review from a team as a code owner July 13, 2026 15:48
@github-actions github-actions Bot added the ci label Jul 13, 2026
@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Changes

Deterministic wheel dependency installation

Layer / File(s) Summary
Compile wheel dependency locks
script/compile-wheel-locks.sh
Adds a script that derives universal, hash-pinned locks from built wheel metadata for the services and plugin variants.
Commit generated dependency locks
.github/wheel-locks/*
Adds pinned external dependency lockfiles with hashes, environment markers, and regeneration guidance.
Use locks in CI wheel smoke tests
.github/workflows/ci.yaml
CI installs dependencies from the committed locks, then installs both wheels with --no-deps before running smoke tests.

Suggested reviewers: a2bondar, crookedstorm, tylersbray, marcusds

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title matches the dependency pinning change but incorrectly says uv.lock instead of the new per-wheel lock files. Rename it to mention the committed wheel lock files, e.g. "ci: pin wheel install-test deps with per-wheel locks".
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci-pin-wheel-smoke-test-deps/schapman

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yaml:
- Around line 1092-1095: Update the uv export command in the wheel smoke-test
setup to include the services extra, using --all-extras or the targeted services
extra option. Keep the subsequent uv tool install of "${WHEEL}[services]"
unchanged so the generated constraints cover the same dependencies being
installed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 78feab1c-d2b5-4a6d-83eb-a19b3a1b795c

📥 Commits

Reviewing files that changed from the base of the PR and between 275c8cd and 2d00247.

📒 Files selected for processing (1)
  • .github/workflows/ci.yaml

Comment thread .github/workflows/ci.yaml Outdated

@crookedstorm crookedstorm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like an improvement to me.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 23845/31009 76.9% 61.7%
Integration Tests 13817/29658 46.6% 19.6%

@SandyChapman
SandyChapman requested review from a team as code owners July 13, 2026 17:47

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
script/compile-wheel-locks.sh (1)

33-36: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Guard against multiple wheel matches.

head -1 silently picks an arbitrary wheel if WHEEL_DIR has more than one match (stale build, leftover artifact). Only the "zero matches" case is guarded. A wrong pick means the committed lock is compiled from the wrong wheel's metadata.

♻️ Add multiplicity check
 np_wheel="$(find "${WHEEL_DIR}" -name 'nemo_platform-*.whl' | head -1)"
 pl_wheel="$(find "${WHEEL_DIR}" -name 'nemo_platform_plugin-*.whl' | head -1)"
+np_count="$(find "${WHEEL_DIR}" -name 'nemo_platform-*.whl' | wc -l)"
+pl_count="$(find "${WHEEL_DIR}" -name 'nemo_platform_plugin-*.whl' | wc -l)"
+[[ "${np_count}" -le 1 ]] || { echo "multiple nemo_platform-*.whl in ${WHEEL_DIR}" >&2; exit 1; }
+[[ "${pl_count}" -le 1 ]] || { echo "multiple nemo_platform_plugin-*.whl in ${WHEEL_DIR}" >&2; exit 1; }
 [[ -n "${np_wheel}" ]] || { echo "no nemo_platform-*.whl in ${WHEEL_DIR}" >&2; exit 1; }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@script/compile-wheel-locks.sh` around lines 33 - 36, Update the wheel
discovery checks around np_wheel and pl_wheel to validate that each pattern
matches exactly one file, rather than selecting the first result with head -1.
Fail with an appropriate error when either wheel has zero or multiple matches,
and only proceed to compile locks when both matches are unique.
.github/workflows/ci.yaml (1)

1090-1103: 🧹 Nitpick | 🔵 Trivial

Consider a lock-freshness check.

--no-deps install trusts the committed lock matches the wheel's current requires-dist. If pyproject.toml deps change without re-running compile-wheel-locks.sh, the smoke test won't catch missing/stale deps until a runtime failure. A CI check diffing wheel metadata Requires-Dist against the lock's non-hash package names would catch drift early.

Also applies to: 1119-1126

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yaml around lines 1090 - 1103, The wheel smoke-test
workflow installs dependencies from a committed lock without verifying it
matches the wheel metadata. Add a lock-freshness check before the --no-deps
install in the wheel validation steps, extracting the wheel’s Requires-Dist
entries and comparing normalized package names against the lock’s non-hash
package names; fail CI on missing or stale dependencies, while preserving the
existing pinned installation flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/ci.yaml:
- Around line 1090-1103: The wheel smoke-test workflow installs dependencies
from a committed lock without verifying it matches the wheel metadata. Add a
lock-freshness check before the --no-deps install in the wheel validation steps,
extracting the wheel’s Requires-Dist entries and comparing normalized package
names against the lock’s non-hash package names; fail CI on missing or stale
dependencies, while preserving the existing pinned installation flow.

In `@script/compile-wheel-locks.sh`:
- Around line 33-36: Update the wheel discovery checks around np_wheel and
pl_wheel to validate that each pattern matches exactly one file, rather than
selecting the first result with head -1. Fail with an appropriate error when
either wheel has zero or multiple matches, and only proceed to compile locks
when both matches are unique.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1a6b9a69-4037-4113-8724-b1c2ff409f75

📥 Commits

Reviewing files that changed from the base of the PR and between f2e909d and 1a513a5.

📒 Files selected for processing (4)
  • .github/wheel-locks/nemo-platform-plugin.txt
  • .github/wheel-locks/nemo-platform-services.txt
  • .github/workflows/ci.yaml
  • script/compile-wheel-locks.sh

The wheel install-smoke-test installed the freshly built nemo-platform /
nemo-platform-plugin wheels by resolving their whole dependency tree fresh from
PyPI (latest-in-range) — non-reproducible and a supply-chain risk (the job then
executes the installed code in the runner). It also pulled litellm 1.92.0, whose
native/PyO3 build has no Python 3.14 wheel, breaking the py3.14 matrix.

Pinning to uv.lock is not possible here: the built vendored-SDK wheel's closure
requires newer deps than the workspace lock (e.g. pydantic>=2.13.3 vs the locked
2.12.5), and a full lock also pins packages that lack py3.14 wheels. Instead,
partial-pin each wheel's DIRECT external deps to a committed constraints file
(.github/wheel-constraints/, regenerated by script/compile-wheel-constraints.sh)
and cap the litellm transitive <1.92; deep transitives resolve normally so they
stay py3.14-compatible and cannot self-conflict. Install both wheels with
`uv pip install --constraint` in a venv.

Drop the litellm cap once litellm ships a Python 3.14 wheel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
@SandyChapman
SandyChapman force-pushed the ci-pin-wheel-smoke-test-deps/schapman branch from 0a28805 to d7c6062 Compare July 13, 2026 19:01
@SandyChapman
SandyChapman enabled auto-merge July 13, 2026 19:04
@SandyChapman
SandyChapman added this pull request to the merge queue Jul 13, 2026
Merged via the queue into main with commit 6018313 Jul 13, 2026
55 checks passed
@SandyChapman
SandyChapman deleted the ci-pin-wheel-smoke-test-deps/schapman branch July 13, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants