Skip to content

fix(rlix): release train GPUs after sync failure#13

Merged
TianyeGGBond merged 1 commit into
rlops:zhenyu/miles-mvp-e2efrom
TianyeGGBond:tianye/m11-after-training-release-finally
May 18, 2026
Merged

fix(rlix): release train GPUs after sync failure#13
TianyeGGBond merged 1 commit into
rlops:zhenyu/miles-mvp-e2efrom
TianyeGGBond:tianye/m11-after-training-release-finally

Conversation

@TianyeGGBond

@TianyeGGBond TianyeGGBond commented May 17, 2026

Copy link
Copy Markdown
Collaborator

What

Wraps sync_base_weights_to_active() in _after_training() with try/finally so the actor_train allocation is released even if weight sync fails after the train actor has already offloaded.

Why

Before:

sync_base_weights_to_active()
_notify_release_cluster_gpus()

If sync raised, _notify_release_cluster_gpus() was skipped and the scheduler could keep actor_train marked as allocated until shutdown_hard.

In multi-pipeline runs, that can block peer pipelines from stepping.

After

try:
    sync_base_weights_to_active()
finally:
    _notify_release_cluster_gpus()

The original sync exception still propagates. This only prevents leaking the scheduler allocation.

Tests

python -m pytest -q tests/test_miles_pipeline_after_training_cleanup.py
1 passed in 0.03s

Scope

Exception-safety hardening only. No happy-path behavior change.

@taoluo

taoluo commented May 17, 2026

Copy link
Copy Markdown
Contributor

lgtm

@TianyeGGBond
TianyeGGBond merged commit a8460a9 into rlops:zhenyu/miles-mvp-e2e May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants