Performance drop on >= 8 nodes (32 GH200) for NCCL >= 2.20

[teojgo]

Each node on the system has 4 GH200 connected with NVLink and communication between nodes happens via the HPE Slingshot Interconnect. The aws-ofi-nccl plugin is used in order to take advantage of the Slingshot Interconnect. Various versions of the plugin have been tried in order to make sure that the performance drop is not related to it.

For example, on 8 nodes (32 GH200) the performance of the sendrecv_perf nccl-test (using 4 tasks per node), where each task is executed using: sendrecv_perf -g 1 -b 512M -e 512M is the following for NCCL 2.19.4:

# Avg bus bandwidth    : 24.2382 

While for NCCL 2.20.5, it is:

# Avg bus bandwidth    : 18.3153 

The numbers above are in GB/s. Up to 4 nodes (16 GH200), the performance is the same.

[sjeaugey]

What is the performance you see on 2 or 4 nodes?
Also, does setting NCCL_NCHANNELS_PER_NET_PEER=4 improve the sendrecv performance?

[teojgo]

@sjeaugey thank you for the suggestion. Setting NCCL_NCHANNELS_PER_NET_PEER=4 gives the expected performance 24.2-24.3 GB/s which is also the one I get with 2 and 4 nodes. What does the variable actually do, because I could not find it in the documentation?

[sjeaugey]

NCCL send/recv operations are currently optimized for alltoall, meaning the send/recv of a pair of ranks will be handled by two CUDA CTAs at most. Setting that variable to 4 means using 4 CTAs per peer, which can help when your communication patterns are sparse, i.e. each rank only communicates with a limited number of peers. It may degrade the alltoall performance though, as each send/recv will be split into 4 parts, while in the alltoall case we have more than enough send/recv operations to saturate the NIC bandwidth, and using just one SM per peer can be enough as you scale.

That said, sendrecv_perf being a ring, I'm not sure why the behavior would change at 8 nodes vs 4. Maybe there is some tuning inside NCCL which changes the number of channels per peer we use as we scale.

[teojgo]

NCCL send/recv operations are currently optimized for alltoall, meaning the send/recv of a pair of ranks will be handled by two CUDA CTAs at most. Setting that variable to 4 means using 4 CTAs per peer, which can help when your communication patterns are sparse, i.e. each rank only communicates with a limited number of peers. It may degrade the alltoall performance though, as each send/recv will be split into 4 parts, while in the alltoall case we have more than enough send/recv operations to saturate the NIC bandwidth, and using just one SM per peer can be enough as you scale.

That said, sendrecv_perf being a ring, I'm not sure why the behavior would change at 8 nodes vs 4. Maybe there is some tuning inside NCCL which changes the number of channels per peer we use as we scale.

Thank you very much. Now it's clear and I think you can close the issue now.