[2.7] Avoid self-message deadlock for local swarm result submission

[pcnudde]

Summary

• avoid synchronous self-message path when trainer submits learn result to itself (aggr == self.me)
• process local submission via _process_learn_result with local peer context, while keeping remote path unchanged
• add unit coverage to verify local self-aggregation submission does not call broadcast_and_wait

Problem

PR #4141 fixed self-message deadlock in _scatter, but result submission in do_learn_task still used broadcast_and_wait(targets=[aggr]). When aggr == self.me with tensor streaming enabled, this can deadlock in synchronous self-message processing.

Test Plan

• added focused unit test in tests/unit_test/app_common/ccwf/test_swarm_self_message_deadlock.py
• validated syntax locally for modified files
• full pytest not run in this environment (pytest not available)

[pcnudde]

/build

[greptile-apps]

Greptile Overview

Greptile Summary

Fixes deadlock in swarm learning when a trainer submits results to itself as aggregator with tensor streaming enabled.

Key Changes:

• Modified do_learn_task in swarm_client_ctl.py to detect self-aggregation (aggr == self.me) and route locally via _process_learn_result instead of broadcast_and_wait
• Creates cloned FL context with peer context to maintain proper execution semantics for local processing
• Keeps remote submission path unchanged for non-self aggregators
• Adds focused unit test verifying broadcast_and_wait is not called for local self-aggregation

Problem Addressed:
When aggr == self.me with tensor streaming enabled, broadcast_and_wait processes self-messages synchronously on the same thread (via CoreCell._send_direct_message). If the message handler calls TensorStreamer.wait_for_tensors(), the thread blocks waiting for data that can only arrive on the same (now blocked) thread, causing deadlock.

Solution Approach:
Similar to PR #4141's fix for _scatter, this PR avoids the synchronous self-message path entirely by calling _process_learn_result directly with a local peer context, while keeping the permission request mechanism unchanged (it doesn't trigger blocking operations).

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The fix follows the exact same pattern as PR [2.7] Fix swarm controller + tensor streaming issue #4141's deadlock fix in _scatter, properly handles self-aggregation by avoiding synchronous self-messages, includes comprehensive unit test coverage, and makes minimal changes isolated to the specific deadlock scenario
• No files require special attention

Important Files Changed

Filename	Overview
nvflare/app_common/ccwf/swarm_client_ctl.py	Adds local self-aggregation path to avoid broadcast_and_wait deadlock when aggr == self.me
tests/unit_test/app_common/ccwf/test_swarm_self_message_deadlock.py	Comprehensive unit test verifying local submission path bypasses broadcast_and_wait for self-aggregation

Sequence Diagram

sequenceDiagram
    participant Trainer as Trainer (site-1)
    participant Aggregator as Aggregator (site-1)
    participant Engine as FL Engine
    participant Gatherer as Result Gatherer

    Note over Trainer,Aggregator: Case: aggr == self.me (Self-Aggregation)
    
    Trainer->>Engine: send_aux_request(request_to_submit)
    Note right of Engine: Permission request (safe - no blocking)
    Engine->>Aggregator: _process_submission_request
    Aggregator->>Gatherer: can_accept_submission()
    Gatherer-->>Aggregator: OK
    Aggregator-->>Engine: make_reply(OK)
    Engine-->>Trainer: Permission granted
    
    alt OLD PATH (Deadlock Risk)
        Trainer->>Engine: broadcast_and_wait(result)
        Note right of Engine: Synchronous self-message
        Engine->>Trainer: _process_learn_result (SAME THREAD)
        Note over Trainer: DEADLOCK if TensorStreamer blocks
    end
    
    alt NEW PATH (Fixed)
        Trainer->>Trainer: Clone fl_ctx + set_peer_context
        Trainer->>Trainer: _process_learn_result(result, local_fl_ctx)
        Note right of Trainer: Direct local call - NO network
        Trainer->>Gatherer: gather(result)
        Gatherer-->>Trainer: make_reply(OK)
    end

Loading

_{Last reviewed commit: de48932}

[greptile-apps]

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}