fix: HTTP2 nil pointer panic and broken GOMEMLIMIT under hostPID#240
Merged
Conversation
Two fixes found during production health check across 3 clusters: 1. HTTP2 nil pointer dereference (ebpftracer/l7/http2.go): When maxActiveRequests limit is hit, the break exits the switch but decoder.Write() still runs with a stale emit func from a previous call, dereferencing a nil request. Set a no-op emit func before breaking so HPACK dynamic table stays in sync without the panic. This was causing 100K+ recovered panics per 6h across KP and DEV, wasting CPU and inflating page cache by ~150Mi per affected pod. 2. GOMEMLIMIT never set under hostPID (main.go): readCgroupMemoryLimit() reads /sys/fs/cgroup/memory.max which doesn't exist under the agent's init.scope cgroup (hostPID places PID 1 at the host root). The pod's actual 1Gi limit lives at /sys/fs/cgroup/kubepods.slice/.../cri-containerd-xxx.scope/memory.max. Fix: read /proc/self/cgroup to resolve the container's own cgroup path. Without this, GOMEMLIMIT was dead code on all clusters — GC didn't trigger during ELF parsing spikes, letting RSS hit 1Gi and causing OOM kills.
There was a problem hiding this comment.
Code Review
This pull request improves HTTP/2 header decoding by ensuring the HPACK dynamic table remains synchronized when the maximum number of active requests is reached. Additionally, it updates the memory limit detection logic to correctly identify container limits in hostPID environments by parsing /proc/self/cgroup. Feedback was provided to extend the memory limit detection to support cgroup v1 in addition to v2 for better compatibility across different Kubernetes environments.
RamanKharchee
previously approved these changes
Apr 13, 2026
Parse /proc/self/cgroup for both v2 (hierarchy-id "0") and v1
("memory" controller) to resolve the container's own cgroup path.
Previously only v2 was handled via /proc/self/cgroup; v1 clusters
with hostPID would still fall through to the root path which reports
the host limit, not the pod limit.
blue4209211
approved these changes
Apr 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes found during production health check across all 3 clusters (KP, KR, DEV):
HTTP2 nil pointer panic: When
maxActiveRequests(100) limit is hit indecodeHeaderBlock(),breakexits the switch butdecoder.Write()still runs with a stale emit func from a previous call, dereferencing a nilreq. Fix: set a no-op emit func before breaking so HPACK dynamic table stays in sync without the panic. This was causing 100K+ recovered panics per 6h across KP and DEV, wasting CPU and inflating page cache by ~150Mi per affected pod from log spam.GOMEMLIMIT never set under hostPID:
readCgroupMemoryLimit()reads/sys/fs/cgroup/memory.maxwhich doesn't exist under the agent'sinit.scopecgroup (hostPID places PID 1 at the host root). The pod's actual 1Gi limit lives at/sys/fs/cgroup/kubepods.slice/.../cri-containerd-xxx.scope/memory.max. Fix: read/proc/self/cgroupto resolve the container's own cgroup path. GOMEMLIMIT was dead code on all 3 clusters — verified viakubectl execthatmemory.maxatinit.scopedoesn't exist. Without GOMEMLIMIT, GC doesn't trigger during ELF parsing spikes (50-100MB per Go binary), letting RSS hit 1Gi → OOM kills.Evidence
Cgroup verification on all clusters:
Test plan
gofmt -l .passes (checked locally)