fix(ci): refresh wandb in the CUDA test container setup - #9789
Merged
Conversation
The framework install pulls protobuf 7.x into an image whose baked wandb predates protobuf 7 support, so `import wandb` dies with "cannot import name ... from wandb_telemetry_pb2" and the unittest job fails at import time for every module touching swift.rlhf_trainers.utils. Upgrade wandb so its generated protobuf code matches the installed protobuf.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Contributor
Author
|
Bumping with fresh evidence that this is still biting: citest runs on current PRs are still failing through the same path this week (perf/megatron-channel-loss on 07-26, fsdp2_npu on 07-24), and main's container setup still carries no wandb refresh, so the old image wandb keeps colliding with the protobuf that the framework install pulls in. This PR's fix is one line in the setup script and was verified locally against the same ImportError chain. Happy to re-verify against the newest image if a maintainer can point me at its tag. |
tastelikefeet
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The unittest job has been red on every PR since around July 14. #9744 reported it, and run 29903129982 on the latest main-facing PR shows the same failure: three test modules die at import with
The chain: the CI image bakes an old wandb, and the CUDA setup's
pip install -r requirements/framework.txt -Upulls protobuf 7.35.1 in over it. The baked wandb's generated protobuf code does not work on protobuf 7.x, soimport wandbinsideswift/rlhf_trainers/utils.pyblows up and takestest_ray_gkd,test_multi_teacherandtest_assemble_teacher_topk_logprobsdown with it. Those modules import the utils module for unrelated helpers; wandb is incidental.Reproduced the mechanism locally: wandb 0.18.7 imports fine with protobuf 5.29.6, fails with
cannot import name 'Deprecated' from wandb.proto.wandb_telemetry_pb2once protobuf 7.35.1 is forced in, and wandb 0.28.1 imports cleanly with protobuf 7.35.1. So refreshing wandb in the setup script is enough. NPU CI is green and untouched.Fixes #9744.