Bugfix: SQ overflow

[chenBright]

What problem does this PR solve?

Issue Number: resolve #3132

Problem Summary:

What is changed and the side effects?

Changed:

Side effects:

• Performance effects:

• Breaking backward compatibility:

---

Check List:

• Please make sure your changes are compilable.
• When providing us with a new feature, it is best to add related tests.
• Please follow Contributor Covenant Code of Conduct.

[chenBright]

We encountered the same problem as #3132. This PR's solution is similar to #3132 (comment) and can solve this problem. PTAL @sunce4t @yanglimingcn

[Copilot]

Pull Request Overview

This PR refactors the RDMA window management by splitting the single _window_size variable into two separate variables: _remote_rq_window_size (tracking remote receive queue capacity) and _sq_window_size (tracking local send queue capacity). This provides more granular control over flow control in RDMA operations.

• Split window size tracking into separate remote RQ and local SQ windows
• Added SendType enum to distinguish between different types of send operations
• Enhanced logging to differentiate between client and server handshake completion

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
src/brpc/rdma/rdma_endpoint.h	Declared two new atomic variables _remote_rq_window_size and _sq_window_size to replace the single _window_size variable
src/brpc/rdma/rdma_endpoint.cpp	Refactored window management logic to independently track local SQ and remote RQ capacity; added SendType enum; improved handshake logging; optimized zero-copy receive path

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yanglimingcn]

Why is CQ divided into send_cq and recv_cq?

[chenBright]

We encountered an error: requests were not being sent, causing a large number of client timeouts.

 [E1008]Reached timeout=60000ms @Socket{id=13 fd=1160 addr=xxx:xx} (0x0x7f957c964ec0) rdma info={rdma_state=ON, handshake_state=ESTABLISHED, rdma_remote_rq_window_size=63, rdma_sq_window_size=0, rdma_local_window_capacity=125, rdma_remote_window_capacity=125, rdma_sbuf_head=57, rdma_sbuf_tail=120, rdma_rbuf_head=36, rdma_unacked_rq_wr=0, rdma_received_ack=0, rdma_unsolicited_sent=0, rdma_unsignaled_sq_wr=1, rdma_new_rq_wrs=0, }

From the RDMA connection information, we found that because ibv_req_notify_cq was only solicited, send WCs did not generate a CQEs. Without recv CQEs, send WCs could not be polled, so ·_sq_window_size` remained at 0. This is likely the reason why both the client and server are unable to send messages.

Using ibv_req_notify_cq with solicited_only=0 could solve this problem, but it would generate too many events. Therefore, we split the CQ into send_cq(solicited_only=0) and recv_cq(solicited_only=1).

[Copilot]

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yanglimingcn]

We encountered an error: requests were not being sent, causing a large number of client timeouts.

 [E1008]Reached timeout=60000ms @Socket{id=13 fd=1160 addr=xxx:xx} (0x0x7f957c964ec0) rdma info={rdma_state=ON, handshake_state=ESTABLISHED, rdma_remote_rq_window_size=63, rdma_sq_window_size=0, rdma_local_window_capacity=125, rdma_remote_window_capacity=125, rdma_sbuf_head=57, rdma_sbuf_tail=120, rdma_rbuf_head=36, rdma_unacked_rq_wr=0, rdma_received_ack=0, rdma_unsolicited_sent=0, rdma_unsignaled_sq_wr=1, rdma_new_rq_wrs=0, }

From the RDMA connection information, we found that because ibv_req_notify_cq was only solicited, send WCs did not generate a CQEs. Without recv CQEs, send WCs could not be polled, so ·_sq_window_size` remained at 0. This is likely the reason why both the client and server are unable to send messages.

Using ibv_req_notify_cq with solicited_only=0 could solve this problem, but it would generate too many events. Therefore, we split the CQ into send_cq(solicited_only=0) and recv_cq(solicited_only=1).

With this modification, send_cq will generate one CQE for every 1/4 of the window？

[chenBright]

With this modification, send_cq will generate one CQE for every 1/4 of the window？

Yes.

[legionxiong]

We encountered an error: requests were not being sent, causing a large number of client timeouts.

 [E1008]Reached timeout=60000ms @Socket{id=13 fd=1160 addr=xxx:xx} (0x0x7f957c964ec0) rdma info={rdma_state=ON, handshake_state=ESTABLISHED, rdma_remote_rq_window_size=63, rdma_sq_window_size=0, rdma_local_window_capacity=125, rdma_remote_window_capacity=125, rdma_sbuf_head=57, rdma_sbuf_tail=120, rdma_rbuf_head=36, rdma_unacked_rq_wr=0, rdma_received_ack=0, rdma_unsolicited_sent=0, rdma_unsignaled_sq_wr=1, rdma_new_rq_wrs=0, }

From the RDMA connection information, we found that because ibv_req_notify_cq was only solicited, send WCs did not generate a CQEs. Without recv CQEs, send WCs could not be polled, so ·_sq_window_size` remained at 0. This is likely the reason why both the client and server are unable to send messages.

Using ibv_req_notify_cq with solicited_only=0 could solve this problem, but it would generate too many events. Therefore, we split the CQ into send_cq(solicited_only=0) and recv_cq(solicited_only=1).

It is unnecessary to split CQ into send_cq and recv_cq, the reason that sliding window goes wrong is the precondition it relies on is not guaranteed in RoCE environment. The sliding window mechanism presumes that an local app receives the ack from remote app means that the underlying SQ has already been released, in a lossless IB environment it maybe true. But in RoCE environment, it is not guaranteed. Because the local device releases SQ only if it receives the ack to a message from remote device, but the ack might be lost and the remote side has processed the message, which means the remote app has received the message and sent an application layer ack to local app. Unfortunately, the local app layer received the ack to a message from remote app and then increased the window, but the device layer is still waiting for the ack from remote device to release the SQ. So the story is the window being increased wrongly for one or two times, till the window is absolutely wrong.

[chenBright]

Unfortunately, the local app layer received the ack to a message from remote app and then increased the window, but the device layer is still waiting for the ack from remote device to release the SQ.

@legionxiong f the device-level ACK is lost, is it impossible to release the SQ?

It is unnecessary to split CQ into send_cq and recv_cq, the reason that sliding window goes wrong is the precondition it relies on is not guaranteed in RoCE environment.

Is there a better solution?

[legionxiong]

the device-level ACK is lost

Local side will resend the message whose ACK was not received within rdma_ack_timeout, and remote side will do nothing but resend the ACK to the message that has been received before. Maybe we need another variable which checks the event of IBV_WC_SEND, it is guaranteed that SQs are released if we polled the work completion of IBV_WC_SEND. And the number of unsignaled SQs could be carried with wr_id.

[chenBright]

Maybe we need another variable which checks the event of IBV_WC_SEND, it is guaranteed that SQs are released if we polled the work completion of IBV_WC_SEND.

[E1008]Reached timeout=60000ms @socket{id=13 fd=1160 addr=xxx:xx} (0x0x7f957c964ec0) rdma info={rdma_state=ON, handshake_state=ESTABLISHED, rdma_remote_rq_window_size=63, rdma_sq_window_size=0, rdma_local_window_capacity=125, rdma_remote_window_capacity=125, rdma_sbuf_head=57, rdma_sbuf_tail=120, rdma_rbuf_head=36, rdma_unacked_rq_wr=0, rdma_received_ack=0, rdma_unsolicited_sent=0, rdma_unsignaled_sq_wr=1, rdma_new_rq_wrs=0, }

In this case, it seems we cannot poll any work completion of IBV_WC_SEND for a long time. With solicited_only=1, if device layer ACKs follows all application layer ACKs, we may be unable to poll IBV_WC_SEND after polling IBV_WC_RECV because there are no CQEs.

Maybe we need another variable which checks the event of IBV_WC_SEND

How can we check for the event of IBV_WC_SEND？

the number of unsignaled SQs could be carried with wr_id.

This solution for updating the SQ window is better.

[randomkang]

I run three tasks, they lasts for 1742、1551、1852 minutes, and the error of "Fail to ibv_post_send: Cannot allocate memory" does not happpen again. @yanglimingcn @chenBright

[yanglimingcn]

LGTM

[chenBright]

I run three tasks, they lasts for 1742、1551、1852 minutes, and the error of "Fail to ibv_post_send: Cannot allocate memory" does not happpen again. @yanglimingcn @chenBright

@randomkang Is this using the latest commit?

[randomkang]

yes, i use the commit of "0794a476ee88e19f91cb9ce43166b4d2f90077b3"

[yanglimingcn]

In fact, I still don't understand the specific function of the 5/4 window in the latest Premiere Pro.

[chenBright]

In fact, I still don't understand the specific function of the 5/4 window in the latest Premiere Pro.

I think it would be more reasonable to add a sliding window to limit the number of IMM.

@randomkang Please try the latest PR again

[yanglimingcn]

So how should this window be set up reasonably? It seems the number of imm items generated is not quite certain?

[chenBright]

So how should this window be set up reasonably? It seems the number of imm items generated is not quite certain?

The window size is related to SendAck.

When _remote_window_capacity is divisible by 2, a maximum of 2 imm are required.

When _remote_window_capacity is not divisible by 2, a maximum of 3 imm are required.

Therefore, RESERVED_WR_NUM is a reasonable value.

brpc/src/brpc/rdma/rdma_endpoint.cpp

Lines 518 to 519 in 7cd0c25

	ep->_remote_window_capacity =
	std::min(ep->_rq_size, remote_msg.sq_size) - RESERVED_WR_NUM,

brpc/src/brpc/rdma/rdma_endpoint.cpp

Lines 911 to 916 in 7cd0c25

	int RdmaEndpoint::SendAck(int num) {
	if (_new_rq_wrs.fetch_add(num, butil::memory_order_relaxed) > _remote_window_capacity / 2) {
	return SendImm(_new_rq_wrs.exchange(0, butil::memory_order_relaxed));
	}
	return 0;
	}

[yanglimingcn]

Since RESERVED_WR_NUM is already present, is the 5/4 no longer needed?

[chenBright]

Since RESERVED_WR_NUM is already present, is the 5/4 no longer needed?

Yes. The capacity of RESERVED_WR_NUM is sufficient; by using the imm window to limit the number of imm elements, 5/4 is no longer needed.

[randomkang]

In fact, I still don't understand the specific function of the 5/4 window in the latest Premiere Pro.

I think it would be more reasonable to add a sliding window to limit the number of IMM.

@randomkang Please try the latest PR again

I run 2 tasks, and the error of "Fail to ibv_post_send: Cannot allocate memory" does not happpen again.

One last for 1790 mins, and fail due to "[wk-15] Thread E1217 16:32:30.900705 106575 12820477405150 external/brpc/src/brpc/rdma/rdma_helper.cpp:220 BlockAllocate] Fail to get block from memory pool". This error is not related to communication.

The other last for 1787mins, and fail due to "[tf-ps-7] W1217 04:18:39.676858 24875 78292958891373 tensorflow/contrib/pms/rpc/brpc_remote_worker.cc:383 operator()] Fail to recv tensorflow.RecvTensorResponse from 10.28.132.10:3642 : Aborted: [E10010][10.28.132.10:3642][E10010]Step 60851325530873895". I don't see the error message of brpc, maybe this error is caused by tensorflow. @chenBright

[randomkang]

I run three tasks, they lasts for 1742、1551、1852 minutes, and the error of "Fail to ibv_post_send: Cannot allocate memory" does not happpen again. @yanglimingcn @chenBright

By the way, these three tasks also failed in final.

The first task failed due to "[wk-8] E1204 21:01:41.101131 72310 62642098032795 external/brpc/src/brpc/rdma/block_pool.cpp:362 AllocBlockFrom] Fail to extend new region. You can set the size of memory pool larger. Refer to the help message of these flags: rdma_memory_pool_initial_size_mb, rdma_memory_pool_increase_size_mb, rdma_memory_pool_max_regions." This error is not related to communication.

The second task failed, but i don't see any error message. Maybe it is killed by the matchine failure.

The third task failed due to "tf op is stuck".

[chenBright]

@randomkang In summary, the error of "Fail to ibv_post_send: Cannot allocate memory" has been resolved. Right?

[randomkang]

@randomkang In summary, the error of "Fail to ibv_post_send: Cannot allocate memory" has been resolved. Right?

Yes, i think so.

[chenBright]

@yanglimingcn Please review it again.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (1)

src/brpc/rdma/rdma_endpoint.cpp:1520

• After setting notified = true and requesting notifications for both CQs at lines 1497-1500, the code continues with the next iteration of the loop. However, at line 1520, notified is unconditionally reset to false when any completion events are found (cnt > 0). This means that if events arrive after notification but are found in the first poll, the notification state is incorrectly reset. This could lead to missing the re-notification step in subsequent iterations when no events are found.

                notified = true;
                continue;
            }
            if (!m->MoreReadEvents(&progress)) {
                break;
            }

            if (0 != ep->GetAndAckEvents(s)) {
                return;
            }

            // Restart polling from `recv_cq'.
            send = false;
            cq = ep->_resource->recv_cq;
            notified = false;
            continue;
        }
        notified = false;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yanglimingcn]

LGTM