[None][chore] Add per-rank routing/scheduling observability to the iteration log - #16323
[None][chore] Add per-rank routing/scheduling observability to the iteration log#16323lancelly wants to merge 2 commits into
Conversation
8ee4509 to
d911ae2
Compare
58ebf5a to
78fc93f
Compare
lancelly
changed the title
…ration log Diagnosing attention-DP load imbalance today requires reconstructing state from per-request metrics. Add per-rank routing and scheduling fields to the existing print_iter_log iteration line: num_active = 14, num_fetched = 3, num_dummy = 1, num_unscheduled = 5 - num_fetched: new requests the ADP router assigned to this rank during this iteration (delta of the cumulative fetch counter). - num_dummy: attention-DP pad dummies in this iteration's scheduled batch (0/1 per rank, padded in when the rank would otherwise run an empty batch). num_scheduled_requests counts the pad, so subtract num_dummy to recover the real scheduled count. - num_unscheduled: active requests not placed in this iteration's batch (len(active_requests) - scheduled batch_size). Sampled at schedule time, so the pad cancels in both terms and it never mixes epochs the way active - num_scheduled would (the num_scheduled printed on this line is the previous iteration's). num_dummy and num_unscheduled are captured at schedule time and so lag one iteration behind num_active/num_fetched, consistent with num_scheduled_requests. No collectives and no new sync points; all fields are log-only. These signals surfaced a 1.8-3.4x per-rank queue-depth skew on an agentic multi-turn workload, motivating the least_queued placement option in NVIDIA#16294. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
78fc93f to
fc514df
Compare
lancelly
changed the title
📝 WalkthroughWalkthroughPyExecutor now tracks dummy, unscheduled, active, and fetched requests per iteration, and includes these counters in profiler debug output. ChangesExecutor iteration metrics
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/bot run --disable-fail-fast |
|
PR_Github #61238 [ run ] triggered by Bot. Commit: |
|
PR_Github #61238 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #61296 [ run ] triggered by Bot. Commit: |
|
PR_Github #61296 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #61503 [ run ] triggered by Bot. Commit: |
|
PR_Github #61503 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #61547 [ run ] triggered by Bot. Commit: |
|
PR_Github #61547 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #61690 [ run ] triggered by Bot. Commit: |
|
PR_Github #61690 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #62057 [ run ] triggered by Bot. Commit: |
|
PR_Github #62057 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #62079 [ run ] triggered by Bot. Commit: |
|
PR_Github #62079 [ run ] completed with state
|
3 participants
Summary by CodeRabbit
Description
Diagnosing attention-DP load imbalance today requires reconstructing state from per-request perf metrics. This PR adds per-rank routing and scheduling fields to the existing
print_iter_logiteration line. It is log-only and touches a single file (py_executor.py).Per-rank fields on the iteration line:
num_active—len(active_requests)at the iteration boundary (top of the loop, after the previous iteration's completions were removed and before the next fetch).num_fetched— new requests the ADP router assigned to this rank during this iteration (delta of the cumulative fetch counter, tracked in the logging closure).num_dummy— attention-DP pad dummies in this iteration's scheduled batch (0/1 per rank, padded in when the rank would otherwise run an empty batch).num_scheduled_requestsalready counts the pad, so subtractnum_dummyto recover the real scheduled count.num_unscheduled— active requests not placed in this iteration's batch (len(active_requests) − scheduled batch_size). Sampled at schedule time, so the pad cancels in both terms and it never mixes epochs the wayactive − num_scheduledwould (thenum_scheduled_requestsprinted on this line is the previous iteration's).num_dummyandnum_unscheduledare captured wherenum_scheduled_requestsis already set (both executor-loop variants), and therefore lag one iteration behindnum_active/num_fetched, consistent withnum_scheduled_requests.