fix: cap HTTP/2 DATA payload accumulation to prevent OOM#226
Merged
Conversation
Http2Parser.Parse appends DATA frame payloads to RequestPayload and ResponsePayload with no size limit. On nodes with high-throughput HTTP/2 traffic (envoy-gateway, gRPC proxies, benchmark-server), these grow unbounded — confirmed via pprof: 149MB (62% of heap) consumed by Http2Parser on a pod that OOMs after ~22h. Cap both payloads at 128KB per stream. They are only used for LLM provider detection and trace spans which need at most a few KB. END_STREAM tracking is unaffected so stream completion still works.
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
RamanKharchee
approved these changes
Mar 13, 2026
mayankpande88
added a commit
that referenced
this pull request
Apr 9, 2026
* fix: update logparser to v0.0.0-20260310062405-9d6258bdd771 Picks up fix/json-log-pattern-hashing (PR #16) which stabilizes JSON log pattern hashing by extracting only message fields. * fix: configurable sensitive log detection with optimized logparser (#224) * fix: update logparser and add configurable sensitive log detection flags - Update logparser to v0.0.0-20260313095946-b22168b95d9c with optimized sensitive parsing (anchor pre-filter, confidence tiers, singleton cache) - Add SENSITIVE_LOG_SAMPLE_RATE, SENSITIVE_LOG_MIN_CONFIDENCE, SENSITIVE_LOG_MAX_DETECTIONS flags for fine-grained control - Update all NewParser callsites to use SensitiveConfig struct - Remove local replace directive * fix: deduplicate SensitiveConfig and align flag naming - Extract SensitiveConfig into a single variable reused across all 3 NewParser calls - Rename flag/env var to sensitive-log-max-detections-per-container / SENSITIVE_LOG_MAX_DETECTIONS_PER_CONTAINER to match variable name * fix: cap HTTP/2 DATA payload accumulation to prevent OOM (#226) Http2Parser.Parse appends DATA frame payloads to RequestPayload and ResponsePayload with no size limit. On nodes with high-throughput HTTP/2 traffic (envoy-gateway, gRPC proxies, benchmark-server), these grow unbounded — confirmed via pprof: 149MB (62% of heap) consumed by Http2Parser on a pod that OOMs after ~22h. Cap both payloads at 128KB per stream. They are only used for LLM provider detection and trace spans which need at most a few KB. END_STREAM tracking is unaffected so stream completion still works. * fix: graceful shutdown, L7 panic protection, and bounded strippedGoExeCache (#225) - Replace os.Exit(0) in signal handler with context cancellation so deferred cleanup (cr.Close, profiling.Stop, resolver.StopWatching) runs on shutdown. Previously leaked eBPF programs, uprobes, and perf buffers on every restart. Use http.Server.Shutdown for graceful HTTP server termination. (fixes #207) - Wrap L7 event processing with recover() to prevent a single malformed packet from crashing the entire agent. Covers both processL7Event and pending L7 retry paths. (fixes #210) - Replace unbounded sync.Map with LRU cache (cap 1000) for strippedGoExeCache to prevent slow memory growth on CI/CD nodes with many unique Go binaries. hashicorp/golang-lru is already a dependency. (fixes #214) * fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes (#227) * fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes HTTP/2 frame parsing was consuming 70% of CPU (profiled). Most HTTP/2 traffic (gRPC, K8s API, service mesh) doesn't need payload capture — only status codes for L7 metrics. - Add lightweight mode to Http2Parser: skips DATA payload accumulation and RequestHeaders map allocation for non-LLM traffic - Auto-upgrade to full mode when :authority matches an LLM provider - Reuse statuses/grpcStatuses maps across Parse() calls (clear instead of alloc) — eliminates ~15% CPU from per-call map allocation - Cap activeRequests at 100 per connection to prevent orphan stream growth - Cap HTTP/2 parsers at 50 per container - Fix parser GC leak: connectionless parsers now cleaned up when pid dies - Eliminate duplicate GetActiveStreamsForLLM() call per event - Skip LLM stream tracker entirely when parser is lightweight * fix: address PR review — GC leak for closed connections, deduplicate END_STREAM - Fix parser GC: closed connection parsers in long-running processes were not cleaned up. Now uses heuristic: if pid is alive but no connection exists, clean up parsers with zero active requests and no partial data. - Add HasPartialData() to Http2Parser for safe GC decisions. - Deduplicate END_STREAM handling: extract it before lightweight/full branch instead of duplicating in both paths. * fix: skip-set for non-LLM HTTP/2 traffic to eliminate CPU waste processHTTP2WithoutConnection produces zero L7 metrics (no connection = no DestinationKey). Its only purpose is LLM detection, yet it was parsing every HTTP/2 frame from gRPC/K8s API traffic — consuming 70% of CPU on busy nodes. Two-phase skip-set approach: 1. If dstIP is known, check DNS cache — skip immediately if not LLM. 2. Otherwise, parse first event per pid:fd in lightweight mode to extract :authority. If not LLM, add to skip-set and never parse that fd again. Also adds GC for skip-set entries when the pid dies. * fix: prevent runtime crash from uint64-to-int overflow in event readers (#228) * fix: prevent runtime crash from uint64-to-int overflow in event readers Both perf and ring buffer L7 event readers convert eBPF-reported payload sizes from uint64 to int without safe clamping. If eBPF sends corrupted data with high-bit-set sizes (e.g. 0x8000000000000001), int() produces a negative value that bypasses the MaxPayloadSize cap, causing make([]byte, negative) to crash the reader goroutine. Also upgrades cilium/ebpf v0.20.0 → v0.21.0 which includes ring buffer and memory management fixes relevant to Go 1.24. Fixes runtime crashes: "fatal error: stack not a power of 2" observed on 12/19 pods after ~7 hours of operation. * fix: revert cilium/ebpf upgrade, keep clampSize fix only cilium/ebpf v0.21.0 has breaking API changes (removed Sym, RewriteConstants, MapName) that affect transitive dependencies. Revert to v0.20.0 and keep just the uint64-to-int overflow fix. * fix: evict stale CounterVec label combinations to prevent series explosion (#231) CounterVec entries for closed TCP connections were never removed, causing series count to grow linearly with pod uptime. On a 40h pod, 89% of emitted counter series were stale (85/95 destinations in a single container had no active connection). At customer scale this accumulated to 989K series (54% of all Prometheus series). Track label combinations pushed to each CounterVec and periodically delete entries whose destinations are no longer in activeConnections. Eviction runs every gcInterval (5min) from the event handler goroutine, taking mu.Lock to exclude concurrent collect() calls. * chore: main to test (#237) * deps: update logparser with structured log level detection (#235) Updates github.com/nudgebee/logparser to include single-pass structured log parsing that extracts level from JSON/logfmt fields instead of unreliable text scanning (nudgebee/logparser#18). Fixes false positive log level classification where INFO logs containing "error"/"fatal" in message content were misclassified as ERROR/CRITICAL. * chore(deps): bump google.golang.org/grpc from 1.76.0 to 1.79.3 (#229) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.76.0 to 1.79.3. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.76.0...v1.79.3) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.79.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: prevent OOM crash from L7 protocol misidentification (#236) The ClickHouse eBPF detector matched non-ClickHouse TCP payloads using a 3-byte heuristic (0x01, 0x00, 0x01), causing ch-go to decode garbage varint lengths and attempt 228TB allocations — a fatal OOM that bypasses defer/recover. 3-layer defense: Layer 0 — eBPF detection hardening: Remove non-port-gated is_clickhouse_query() call. ClickHouse native protocol detection now only on ports 9000/8123. Layer 1 — Bounded parsing: Wrap ch-go proto.NewReader with io.LimitReader(payload size). Varint lengths exceeding actual payload now return io.EOF instead of OOM. Layer 2 — Userspace validation: Add structural checks before invoking external libraries: - ClickHouse: validate query code byte + query ID length - Postgres: reject unknown frame types (Q/B/P/C only) - Zookeeper: validate opcode in known range Layer 3 — Protocol reclassification: Track consecutive parse failures per connection. After 3 failures, override the cached protocol to stop further misidentified parsing. Logs a warning for observability. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Shiv <3078106+blue4209211@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Http2Parser.Parseappends DATA frame payloads (RequestPayload,ResponsePayload) with no size limit. On nodes with high-throughput HTTP/2 traffic (envoy-gateway, gRPC proxies), these grow unbounded causing OOM.Test plan