Skip to content

fix: HTTP2 nil pointer panic and broken GOMEMLIMIT under hostPID#240

Merged
blue4209211 merged 2 commits into
mainfrom
fix/http2-nil-panic-and-cgroup-memlimit
Apr 13, 2026
Merged

fix: HTTP2 nil pointer panic and broken GOMEMLIMIT under hostPID#240
blue4209211 merged 2 commits into
mainfrom
fix/http2-nil-panic-and-cgroup-memlimit

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

Two fixes found during production health check across all 3 clusters (KP, KR, DEV):

  • HTTP2 nil pointer panic: When maxActiveRequests (100) limit is hit in decodeHeaderBlock(), break exits the switch but decoder.Write() still runs with a stale emit func from a previous call, dereferencing a nil req. Fix: set a no-op emit func before breaking so HPACK dynamic table stays in sync without the panic. This was causing 100K+ recovered panics per 6h across KP and DEV, wasting CPU and inflating page cache by ~150Mi per affected pod from log spam.

  • GOMEMLIMIT never set under hostPID: readCgroupMemoryLimit() reads /sys/fs/cgroup/memory.max which doesn't exist under the agent's init.scope cgroup (hostPID places PID 1 at the host root). The pod's actual 1Gi limit lives at /sys/fs/cgroup/kubepods.slice/.../cri-containerd-xxx.scope/memory.max. Fix: read /proc/self/cgroup to resolve the container's own cgroup path. GOMEMLIMIT was dead code on all 3 clusters — verified via kubectl exec that memory.max at init.scope doesn't exist. Without GOMEMLIMIT, GC doesn't trigger during ELF parsing spikes (50-100MB per Go binary), letting RSS hit 1Gi → OOM kills.

Evidence

Cluster OOMKills Panic count (6h) Peak memory
KP (prod) 0 35K (lhbc8), 23K (tgtfk), 13K (vjnz9) -
KR (rackspace) 2 0 (old image) -
DEV (GKE) 2 39K (v6d55), 29K (fh94b) 1024Mi (v6d55, xdmxm, q9pbt)

Cgroup verification on all clusters:

/proc/1/cgroup  → 0::/init.scope          (host root, no memory.max)
/proc/self/cgroup → 0::/kubepods.slice/... (memory.max = 1073741824)

Test plan

  • Verify gofmt -l . passes (checked locally)
  • Deploy to DEV, confirm GOMEMLIMIT log line appears at startup
  • Monitor DEV panic counts — should drop to zero
  • Monitor DEV memory peaks — should stay well below 1Gi
  • After 24h stability on DEV, roll to KP

Two fixes found during production health check across 3 clusters:

1. HTTP2 nil pointer dereference (ebpftracer/l7/http2.go):
   When maxActiveRequests limit is hit, the break exits the switch but
   decoder.Write() still runs with a stale emit func from a previous
   call, dereferencing a nil request. Set a no-op emit func before
   breaking so HPACK dynamic table stays in sync without the panic.
   This was causing 100K+ recovered panics per 6h across KP and DEV,
   wasting CPU and inflating page cache by ~150Mi per affected pod.

2. GOMEMLIMIT never set under hostPID (main.go):
   readCgroupMemoryLimit() reads /sys/fs/cgroup/memory.max which
   doesn't exist under the agent's init.scope cgroup (hostPID places
   PID 1 at the host root). The pod's actual 1Gi limit lives at
   /sys/fs/cgroup/kubepods.slice/.../cri-containerd-xxx.scope/memory.max.
   Fix: read /proc/self/cgroup to resolve the container's own cgroup
   path. Without this, GOMEMLIMIT was dead code on all clusters —
   GC didn't trigger during ELF parsing spikes, letting RSS hit 1Gi
   and causing OOM kills.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves HTTP/2 header decoding by ensuring the HPACK dynamic table remains synchronized when the maximum number of active requests is reached. Additionally, it updates the memory limit detection logic to correctly identify container limits in hostPID environments by parsing /proc/self/cgroup. Feedback was provided to extend the memory limit detection to support cgroup v1 in addition to v2 for better compatibility across different Kubernetes environments.

Comment thread main.go Outdated
RamanKharchee
RamanKharchee previously approved these changes Apr 13, 2026
Parse /proc/self/cgroup for both v2 (hierarchy-id "0") and v1
("memory" controller) to resolve the container's own cgroup path.
Previously only v2 was handled via /proc/self/cgroup; v1 clusters
with hostPID would still fall through to the root path which reports
the host limit, not the pod limit.
@blue4209211 blue4209211 merged commit 9b11a75 into main Apr 13, 2026
1 check passed
@blue4209211 blue4209211 deleted the fix/http2-nil-panic-and-cgroup-memlimit branch April 13, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants