http: emit drain on socket takeover and avoid stale HWM reuse

[trivenay]

When OutgoingMessage transitions from pre-socket buffering (Path B) to socket-connected writing (Path A), the backpressure domain changes. The OM should emit drain at this transition to signal that its buffer is clear and the caller can resume writing under the socket's own backpressure.

Previously, _flush() gated drain emission on writableLength === 0 (which includes socket.writableLength). This conflated the OM's buffer state with the socket's kernel write queue. When the socket had a higher writableHighWaterMark than the OM (e.g., agent reuses a socket from a prior request with a different HWM), the socket was never backpressured, never emitted drain — permanent deadlock.

Approach

This PR makes two changes to address the problem:

1. Drain fix in _flush() (the must-have): Once _flushOutput() completes and all buffered data has been handed to the socket, emit drain unconditionally. From this point, the socket enforces its own backpressure via socket.write() return values. We don't wait for socket.writableLength to reach zero because that's the socket's backpressure domain — not the OM's. If the socket is full, the very next write() through Path A will return false and the user stops writing again naturally.

2. Agent HWM mismatch check (defense in depth): Don't reuse a pooled socket in http.Agent if its writableHighWaterMark differs from the request's highWaterMark. This ensures the user's backpressure threshold is respected for users of the built-in http.Agent. We chose to include this because highWaterMark on a connected TCP socket cannot be changed after creation (the underlying kernel buffer is not exposed via Node's TCP handle, and _writableState.highWaterMark is cosmetic since state.length stays 0 for connected sockets). Since there's no way to make a reused socket respect a different HWM, the most resilient approach is to not reuse it. For requests to the same host:port it's rare that different highWaterMark values are used, so socket reuse still happens for the vast majority of connections.

The drain fix alone prevents the deadlock universally (including custom agents and createConnection). The agent check additionally ensures correct backpressure behavior — not just absence of deadlock — for the common case.

Deadlock reproduction (requires reduced TCP send buffer)

const http = require('http');

const server = http.createServer((req, res) => {
  setTimeout(() => { req.resume(); req.on('end', () => res.end('ok')); }, 30000);
}).listen(0, () => {
  const port = server.address().port;
  const agent = new http.Agent({ keepAlive: true });

  // Request A: creates socket with HWM=10MB
  http.request({ port, method: 'POST', agent, highWaterMark: 10 * 1024 * 1024 }, (res) => {
    res.resume();
    res.on('end', () => {
      setTimeout(() => {
        // Request B: default HWM (64KB), reuses socket (HWM=10MB)
        const req = http.request({ port, method: 'POST', agent });
        // Write 2MB: > 64KB OM HWM, < 10MB socket HWM, > kernel TCP buffer
        const r = req.write(Buffer.alloc(2 * 1024 * 1024));
        if (!r) {
          setTimeout(() => { console.error('DEADLOCK'); process.exit(1); }, 15000);
          req.on('drain', () => req.end());
        } else {
          req.end();
        }
      }, 100);
    });
  }).end('x');
});

sysctl -w net.ipv4.tcp_wmem="4096 16384 65536"
node repro.js  # DEADLOCK without fix, drain fires with fix

Fixes: #64680
Refs: #64653
Refs: #62936

[nodejs-github-bot]

Review requested:

• @nodejs/http
• @nodejs/net

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.29%. Comparing base (f00fb75) to head (94a47d6).
⚠️ Report is 42 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #64991      +/-   ##
==========================================
+ Coverage   90.27%   90.29%   +0.01%     
==========================================
  Files         762      759       -3     
  Lines      247534   247624      +90     
  Branches    46694    46689       -5     
==========================================
+ Hits       223457   223587     +130     
+ Misses      15529    15512      -17     
+ Partials     8548     8525      -23     

Files with missing lines	Coverage Δ
lib/_http_agent.js	96.18% <100.00%> (+0.05%)	⬆️
lib/_http_outgoing.js	97.78% <100.00%> (+0.14%)	⬆️

... and 62 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ronag]

Why not just update the socket hwm?

[trivenay]

@ronag Thanks for the suggestion. My initial thought was that this would be fragile since HWM on a connected socket operates at the TCP/libuv layer and changing it after creation might not reliably affect backpressure behavior. But after testing, it does work — state.length accumulates through the HTTP write path and the HWM check is effective:

# With reduced kernel buffer + HWM synced on reused socket:
sysctl -w net.ipv4.tcp_wmem="4096 4096 16384"

Socket HWM before sync: 10485760
Socket HWM after sync: 65536
write(2MB): false
drain fired — HWM sync works

# Also verified on Path A (socket connected, write after delay):
Before write: state.length= 0 HWM= 10240
After write: state.length= 51302 returned: false
drain fired

Will update the PR to sync the socket HWM instead of destroying it. The drain fix in _flush() is still needed separately for custom agents and createConnection where we can't control the socket's HWM.

[ronag]

I'm a little skeptical but I don't think it's super harmful. The thing here is this.writableLength === 0 basically means that the socket doesn't want more data, and ignoring that will cause us to always over buffer. I'm not sure I understand what this is fixing. Semantically it's wrong...

[ronag]

maybe this.writableLength < hwm?

[trivenay]

This fixes the deadlock from #64680 — the previous fix (#64653) didn't cover cases where the socket comes from agent reuse or custom createConnection with a different HWM that we can't control (detailed repro and analysis here). At the changeover, _flushOutput() moves all bytes from the OM buffer to the socket. The old check requires writableLength === 0 before emitting drain — but after flush, writableLength is just socket.writableLength (OM buffer is empty). If the socket has a higher HWM than the OM, it accepted all the data without backpressure (socket.write() returned true) — so it will never emit drain on its own. And the kernel may not have consumed the bytes yet, leaving socket.writableLength > 0. No one triggers drain → deadlock. We reproduced this with reduced TCP buffer settings and confirmed this change resolves it.

The logic: once Path A (socket) takes over, it should also own backpressure. So we emit drain once all bytes move from OM to socket — the next write goes directly to the socket which will return false if it's full. No over-buffering, just one write attempt that the socket correctly handles.

this.writableLength < hwm wouldn't fix it because after flush, socket.writableLength can exceed hwm (data queued in libuv while kernel is slow), so the condition still fails and drain never fires.

[trivenay]

Happy to discuss further — if you think this.writableLength < hwm is more appropriate for other reasons, we can explore that. My concern is that after flush, socket.writableLength can be larger than hwm while the kernel drains, which would prevent drain from firing and leave us in the same deadlock. But if there's a scenario where unconditional drain causes issues beyond the one-extra-write that Path A immediately backpressures, I would love to understand it.

Also I have addressed your earlier feedback on the agent side — replaced the socket destroy with a HWM. Please have a look when possible.

[ronag]

sounds like there is a bigger problem:

socket.writableLength can be larger than hwm while the kernel drains, which would prevent drain from firing and leave us in the same deadlock.

This sounds wrong, and if this is the case then that's what we should fix. writableLength is allowed to be larger than hwm and drain should fire.

[trivenay]

Thinking about this more — the only remaining case is when a custom agent or createConnection provides a socket with a different HWM that we can't control. With #64653 we synced the OM's kHighWaterMark to the user's value, and in this PR we sync the reused socket's HWM to match. So the only gap is externally-provided sockets.

For that case: _flushOutput() already returns the last socket.write() return value. If it returns false, the socket is backpressured and will emit drain — which ondrain() catches and emits on the request for the user's listener. So we don't need to do anything. If it returns true, the socket won't emit drain on its own — so we need to emit it ourselves.

const ret = this._flushOutput(socket);

if (this.finished) {
  this._finish();
} else if (this[kNeedDrain] && ret !== false) {
  // Socket accepted all data without backpressure — it won't emit
  // drain, so we emit it since the OM buffer is now clear.
  this[kNeedDrain] = false;
  this.emit(drain);
}

This way we only emit drain when the socket genuinely won't do it itself. When the socket IS backpressured, the normal ondrain() path handles it and we don't over-buffer.

If this approach looks right, I'll update the PR with this change.

[ronag]

Sounds much more reasonable

[trivenay]

Thanks — pushed the updated approach. _flush() now captures the return value from _flushOutput(): if ret !== false (socket accepted without backpressure, won't emit drain on its own), we emit drain; otherwise we leave it to ondrain() which handles the backpressured case normally.