Skip to content

Fix out_forward keepalive reuse after remote close - #5309

Merged
Watson1978 merged 1 commit into
fluent:masterfrom
aviralgarg05:fix-forward-keepalive-closed-socket-reuse
Apr 17, 2026
Merged

Fix out_forward keepalive reuse after remote close#5309
Watson1978 merged 1 commit into
fluent:masterfrom
aviralgarg05:fix-forward-keepalive-closed-socket-reuse

Conversation

@aviralgarg05

@aviralgarg05 aviralgarg05 commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Which issue(s) this PR fixes:
Fixes #5269
Fixes #4618

What this PR does / why we need it:
This PR fixes a keepalive socket reuse bug in out_forward.

When a cached keepalive connection has already been closed by the remote side, out_forward could pick that socket back up and try to write to it again. In the reported TLS/NLB setup, that leaves the flush thread spinning on a dead socket and can drive CPU usage to 100%.

The fix is to validate cached sockets before reusing them. If a socket has already become unreadable, closed, or otherwise looks no longer safe to reuse, it is discarded and a new connection is created instead.

This keeps out_forward from reusing half-closed keepalive sockets and avoids the busy-loop behavior described in the issue.

This PR also adds regression coverage to make sure a socket that was closed by the peer is not reused from the keepalive cache.

Docs Changes:
None.

Release Note:

  • out_forward: avoid reusing closed keepalive sockets after remote disconnects

Signed-off-by: aviralgarg05 <gargaviral99@gmail.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Thanks a lot for this fix!

@Watson1978 Watson1978 added this to the v1.20.0 milestone Apr 13, 2026
@Watson1978 Watson1978 added the backport to v1.19 We will backport this fix to the LTS branch label Apr 13, 2026
@Watson1978
Watson1978 requested a review from kenhys April 16, 2026 06:24

@kenhys kenhys left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@Watson1978
Watson1978 merged commit 538b8b6 into fluent:master Apr 17, 2026
21 checks passed
github-actions Bot pushed a commit that referenced this pull request Apr 20, 2026
**Which issue(s) this PR fixes**:
Fixes #5269
Fixes #4618

**What this PR does / why we need it**:
This PR fixes a keepalive socket reuse bug in `out_forward`.

When a cached keepalive connection has already been closed by the remote
side, `out_forward` could pick that socket back up and try to write to
it again. In the reported TLS/NLB setup, that leaves the flush thread
spinning on a dead socket and can drive CPU usage to 100%.

The fix is to validate cached sockets before reusing them. If a socket
has already become unreadable, closed, or otherwise looks no longer safe
to reuse, it is discarded and a new connection is created instead.

This keeps `out_forward` from reusing half-closed keepalive sockets and
avoids the busy-loop behavior described in the issue.

This PR also adds regression coverage to make sure a socket that was
closed by the peer is not reused from the keepalive cache.

**Docs Changes**:
None.

**Release Note**:
- out_forward: avoid reusing closed keepalive sockets after remote
disconnects

Signed-off-by: aviralgarg05 <gargaviral99@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Watson1978 pushed a commit that referenced this pull request Apr 20, 2026
…5309) (#5343)

**Which issue(s) this PR fixes**:  
Fixes #5269
Fixes #4618

**What this PR does / why we need it**:  
This PR fixes a keepalive socket reuse bug in `out_forward`.

When a cached keepalive connection has already been closed by the remote
side, `out_forward` could pick that socket back up and try to write to
it again. In the reported TLS/NLB setup, that leaves the flush thread
spinning on a dead socket and can drive CPU usage to 100%.

The fix is to validate cached sockets before reusing them. If a socket
has already become unreadable, closed, or otherwise looks no longer safe
to reuse, it is discarded and a new connection is created instead.

This keeps `out_forward` from reusing half-closed keepalive sockets and
avoids the busy-loop behavior described in the issue.

This PR also adds regression coverage to make sure a socket that was
closed by the peer is not reused from the keepalive cache.

**Docs Changes**:  
None.

**Release Note**:  
- out_forward: avoid reusing closed keepalive sockets after remote
disconnects

Signed-off-by: aviralgarg05 <gargaviral99@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aviral Garg <gargaviral99@gmail.com>
@Watson1978 Watson1978 added the backported "backport to LTS" is done label Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport to v1.19 We will backport this fix to the LTS branch backported "backport to LTS" is done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Forward plugin 100% CPU spikes with keepalive enabled Output forward connections are in CLOSE_WAIT state

3 participants