fix: prevent runtime crash from uint64-to-int overflow in event readers#228
Merged
Conversation
Both perf and ring buffer L7 event readers convert eBPF-reported payload sizes from uint64 to int without safe clamping. If eBPF sends corrupted data with high-bit-set sizes (e.g. 0x8000000000000001), int() produces a negative value that bypasses the MaxPayloadSize cap, causing make([]byte, negative) to crash the reader goroutine. Also upgrades cilium/ebpf v0.20.0 → v0.21.0 which includes ring buffer and memory management fixes relevant to Go 1.24. Fixes runtime crashes: "fatal error: stack not a power of 2" observed on 12/19 pods after ~7 hours of operation.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical stability issue where Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new clampSize helper function in ebpftracer/tracer.go to safely convert uint64 values to int while capping them at a specified maximum size, preventing potential negative integer conversions. This function is then utilized to refactor and improve the payload and response length clamping logic within the runEventsReader and runRingbufEventsReader functions. Additionally, the PR updates several Go module dependencies, including github.com/cilium/ebpf to v0.21.0, github.com/klauspost/compress, github.com/mdlayher/socket, github.com/opencontainers/image-spec, github.com/spf13/cobra, and github.com/spf13/pflag, along with their corresponding go.sum entries.
cilium/ebpf v0.21.0 has breaking API changes (removed Sym, RewriteConstants, MapName) that affect transitive dependencies. Revert to v0.20.0 and keep just the uint64-to-int overflow fix.
RamanKharchee
approved these changes
Mar 16, 2026
blue4209211
approved these changes
Mar 16, 2026
mayankpande88
added a commit
that referenced
this pull request
Apr 9, 2026
* fix: update logparser to v0.0.0-20260310062405-9d6258bdd771 Picks up fix/json-log-pattern-hashing (PR #16) which stabilizes JSON log pattern hashing by extracting only message fields. * fix: configurable sensitive log detection with optimized logparser (#224) * fix: update logparser and add configurable sensitive log detection flags - Update logparser to v0.0.0-20260313095946-b22168b95d9c with optimized sensitive parsing (anchor pre-filter, confidence tiers, singleton cache) - Add SENSITIVE_LOG_SAMPLE_RATE, SENSITIVE_LOG_MIN_CONFIDENCE, SENSITIVE_LOG_MAX_DETECTIONS flags for fine-grained control - Update all NewParser callsites to use SensitiveConfig struct - Remove local replace directive * fix: deduplicate SensitiveConfig and align flag naming - Extract SensitiveConfig into a single variable reused across all 3 NewParser calls - Rename flag/env var to sensitive-log-max-detections-per-container / SENSITIVE_LOG_MAX_DETECTIONS_PER_CONTAINER to match variable name * fix: cap HTTP/2 DATA payload accumulation to prevent OOM (#226) Http2Parser.Parse appends DATA frame payloads to RequestPayload and ResponsePayload with no size limit. On nodes with high-throughput HTTP/2 traffic (envoy-gateway, gRPC proxies, benchmark-server), these grow unbounded — confirmed via pprof: 149MB (62% of heap) consumed by Http2Parser on a pod that OOMs after ~22h. Cap both payloads at 128KB per stream. They are only used for LLM provider detection and trace spans which need at most a few KB. END_STREAM tracking is unaffected so stream completion still works. * fix: graceful shutdown, L7 panic protection, and bounded strippedGoExeCache (#225) - Replace os.Exit(0) in signal handler with context cancellation so deferred cleanup (cr.Close, profiling.Stop, resolver.StopWatching) runs on shutdown. Previously leaked eBPF programs, uprobes, and perf buffers on every restart. Use http.Server.Shutdown for graceful HTTP server termination. (fixes #207) - Wrap L7 event processing with recover() to prevent a single malformed packet from crashing the entire agent. Covers both processL7Event and pending L7 retry paths. (fixes #210) - Replace unbounded sync.Map with LRU cache (cap 1000) for strippedGoExeCache to prevent slow memory growth on CI/CD nodes with many unique Go binaries. hashicorp/golang-lru is already a dependency. (fixes #214) * fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes (#227) * fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes HTTP/2 frame parsing was consuming 70% of CPU (profiled). Most HTTP/2 traffic (gRPC, K8s API, service mesh) doesn't need payload capture — only status codes for L7 metrics. - Add lightweight mode to Http2Parser: skips DATA payload accumulation and RequestHeaders map allocation for non-LLM traffic - Auto-upgrade to full mode when :authority matches an LLM provider - Reuse statuses/grpcStatuses maps across Parse() calls (clear instead of alloc) — eliminates ~15% CPU from per-call map allocation - Cap activeRequests at 100 per connection to prevent orphan stream growth - Cap HTTP/2 parsers at 50 per container - Fix parser GC leak: connectionless parsers now cleaned up when pid dies - Eliminate duplicate GetActiveStreamsForLLM() call per event - Skip LLM stream tracker entirely when parser is lightweight * fix: address PR review — GC leak for closed connections, deduplicate END_STREAM - Fix parser GC: closed connection parsers in long-running processes were not cleaned up. Now uses heuristic: if pid is alive but no connection exists, clean up parsers with zero active requests and no partial data. - Add HasPartialData() to Http2Parser for safe GC decisions. - Deduplicate END_STREAM handling: extract it before lightweight/full branch instead of duplicating in both paths. * fix: skip-set for non-LLM HTTP/2 traffic to eliminate CPU waste processHTTP2WithoutConnection produces zero L7 metrics (no connection = no DestinationKey). Its only purpose is LLM detection, yet it was parsing every HTTP/2 frame from gRPC/K8s API traffic — consuming 70% of CPU on busy nodes. Two-phase skip-set approach: 1. If dstIP is known, check DNS cache — skip immediately if not LLM. 2. Otherwise, parse first event per pid:fd in lightweight mode to extract :authority. If not LLM, add to skip-set and never parse that fd again. Also adds GC for skip-set entries when the pid dies. * fix: prevent runtime crash from uint64-to-int overflow in event readers (#228) * fix: prevent runtime crash from uint64-to-int overflow in event readers Both perf and ring buffer L7 event readers convert eBPF-reported payload sizes from uint64 to int without safe clamping. If eBPF sends corrupted data with high-bit-set sizes (e.g. 0x8000000000000001), int() produces a negative value that bypasses the MaxPayloadSize cap, causing make([]byte, negative) to crash the reader goroutine. Also upgrades cilium/ebpf v0.20.0 → v0.21.0 which includes ring buffer and memory management fixes relevant to Go 1.24. Fixes runtime crashes: "fatal error: stack not a power of 2" observed on 12/19 pods after ~7 hours of operation. * fix: revert cilium/ebpf upgrade, keep clampSize fix only cilium/ebpf v0.21.0 has breaking API changes (removed Sym, RewriteConstants, MapName) that affect transitive dependencies. Revert to v0.20.0 and keep just the uint64-to-int overflow fix. * fix: evict stale CounterVec label combinations to prevent series explosion (#231) CounterVec entries for closed TCP connections were never removed, causing series count to grow linearly with pod uptime. On a 40h pod, 89% of emitted counter series were stale (85/95 destinations in a single container had no active connection). At customer scale this accumulated to 989K series (54% of all Prometheus series). Track label combinations pushed to each CounterVec and periodically delete entries whose destinations are no longer in activeConnections. Eviction runs every gcInterval (5min) from the event handler goroutine, taking mu.Lock to exclude concurrent collect() calls. * chore: main to test (#237) * deps: update logparser with structured log level detection (#235) Updates github.com/nudgebee/logparser to include single-pass structured log parsing that extracts level from JSON/logfmt fields instead of unreliable text scanning (nudgebee/logparser#18). Fixes false positive log level classification where INFO logs containing "error"/"fatal" in message content were misclassified as ERROR/CRITICAL. * chore(deps): bump google.golang.org/grpc from 1.76.0 to 1.79.3 (#229) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.76.0 to 1.79.3. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.76.0...v1.79.3) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.79.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: prevent OOM crash from L7 protocol misidentification (#236) The ClickHouse eBPF detector matched non-ClickHouse TCP payloads using a 3-byte heuristic (0x01, 0x00, 0x01), causing ch-go to decode garbage varint lengths and attempt 228TB allocations — a fatal OOM that bypasses defer/recover. 3-layer defense: Layer 0 — eBPF detection hardening: Remove non-port-gated is_clickhouse_query() call. ClickHouse native protocol detection now only on ports 9000/8123. Layer 1 — Bounded parsing: Wrap ch-go proto.NewReader with io.LimitReader(payload size). Varint lengths exceeding actual payload now return io.EOF instead of OOM. Layer 2 — Userspace validation: Add structural checks before invoking external libraries: - ClickHouse: validate query code byte + query ID length - Postgres: reject unknown frame types (Q/B/P/C only) - Zookeeper: validate opcode in known range Layer 3 — Protocol reclassification: Track consecutive parse failures per connection. After 3 failures, override the cached protocol to stop further misidentified parsing. Logs a warning for observability. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Shiv <3078106+blue4209211@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
int(uint64)conversion in both perf and ring buffer L7 event readers that can produce negative slice lengths from corrupted eBPF data, crashing the reader goroutineclampSize()helper that safely caps uint64 values before converting to intContext
12/19 node-agent pods were crashing with
fatal error: stack not a power of 2after ~7 hours of operation. The uint64→int overflow is the most likely trigger — corrupted eBPF payload sizes bypass the MaxPayloadSize check and causemake([]byte, negative)which crashes the runtime.Test plan