Skip to content

fix: graceful shutdown, L7 panic protection, bounded cache#225

Merged
mayankpande88 merged 1 commit into
mainfrom
fix/graceful-shutdown
Mar 13, 2026
Merged

fix: graceful shutdown, L7 panic protection, bounded cache#225
mayankpande88 merged 1 commit into
mainfrom
fix/graceful-shutdown

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

Test plan

  • gofmt -l . passes
  • Agent starts and shuts down cleanly on SIGTERM (deferred cleanup runs, logs "shutdown complete")
  • Malformed L7 packets are logged and skipped instead of crashing the agent
  • strippedGoExeCache evicts oldest entries when exceeding 1000

Closes #207, #210, #214

…eCache

- Replace os.Exit(0) in signal handler with context cancellation so deferred
  cleanup (cr.Close, profiling.Stop, resolver.StopWatching) runs on shutdown.
  Previously leaked eBPF programs, uprobes, and perf buffers on every restart.
  Use http.Server.Shutdown for graceful HTTP server termination. (fixes #207)

- Wrap L7 event processing with recover() to prevent a single malformed packet
  from crashing the entire agent. Covers both processL7Event and pending L7
  retry paths. (fixes #210)

- Replace unbounded sync.Map with LRU cache (cap 1000) for strippedGoExeCache
  to prevent slow memory growth on CI/CD nodes with many unique Go binaries.
  hashicorp/golang-lru is already a dependency. (fixes #214)
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the stability, resource management, and operational robustness of the agent. It introduces mechanisms for controlled shutdown, prevents critical failures from isolated L7 processing errors, and optimizes memory usage by bounding a previously unbounded cache. These changes collectively contribute to a more reliable and efficient agent, particularly in dynamic environments.

Highlights

  • Graceful shutdown: Replaced os.Exit(0) with context cancellation to ensure proper cleanup of resources like cr.Close(), profiling.Stop(), and resolver.StopWatching() on SIGTERM, and used http.Server.Shutdown for clean HTTP termination, preventing eBPF program leaks.
  • L7 panic protection: Implemented recover() around processL7Event and pending L7 retries to prevent the agent from crashing due to malformed L7 packets, ensuring continuous observability.
  • Bounded cache: Replaced the unbounded sync.Map for strippedGoExeCache with a hashicorp/golang-lru cache capped at 1000 entries, addressing slow memory growth on CI/CD nodes with numerous short-lived Go binaries.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • containers/registry.go
    • Added recover() to processL7Event to catch panics.
    • Introduced safeOnL7Request function to wrap onL7RequestWithResult with panic recovery for pending L7 retries.
    • Updated processPendingL7Events to use the new safeOnL7Request for safer L7 event processing.
  • ebpftracer/tls.go
    • Replaced sync.Map with hashicorp/golang-lru/v2 for strippedGoExeCache.
    • Updated cache operations from Load/Store to Contains/Add respectively.
    • Removed sync import and added lru import.
  • main.go
    • Imported the context package.
    • Modified signal handling to use context.WithCancel instead of os.Exit(0) for graceful shutdown.
    • Implemented http.Server.Shutdown for controlled HTTP server termination.
    • Added defer resolver.StopWatching() to ensure resolver cleanup.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several important reliability improvements: graceful shutdown handling, panic protection in L7 event processing, and a bounded cache for Go binaries to prevent memory leaks. The changes are well-implemented and address critical issues like resource leakage and agent crashes. I've added one suggestion for main.go to make the signal handling even more robust by stopping signal notifications after the first one is received.

Comment thread main.go
@mayankpande88 mayankpande88 merged commit 655f5e4 into main Mar 13, 2026
1 check passed
@mayankpande88 mayankpande88 deleted the fix/graceful-shutdown branch March 13, 2026 16:34
mayankpande88 added a commit that referenced this pull request Apr 9, 2026
* fix: update logparser to v0.0.0-20260310062405-9d6258bdd771

Picks up fix/json-log-pattern-hashing (PR #16) which stabilizes
JSON log pattern hashing by extracting only message fields.

* fix: configurable sensitive log detection with optimized logparser (#224)

* fix: update logparser and add configurable sensitive log detection flags

- Update logparser to v0.0.0-20260313095946-b22168b95d9c with optimized
  sensitive parsing (anchor pre-filter, confidence tiers, singleton cache)
- Add SENSITIVE_LOG_SAMPLE_RATE, SENSITIVE_LOG_MIN_CONFIDENCE,
  SENSITIVE_LOG_MAX_DETECTIONS flags for fine-grained control
- Update all NewParser callsites to use SensitiveConfig struct
- Remove local replace directive

* fix: deduplicate SensitiveConfig and align flag naming

- Extract SensitiveConfig into a single variable reused across all 3 NewParser calls
- Rename flag/env var to sensitive-log-max-detections-per-container / SENSITIVE_LOG_MAX_DETECTIONS_PER_CONTAINER to match variable name

* fix: cap HTTP/2 DATA payload accumulation to prevent OOM (#226)

Http2Parser.Parse appends DATA frame payloads to RequestPayload and
ResponsePayload with no size limit. On nodes with high-throughput
HTTP/2 traffic (envoy-gateway, gRPC proxies, benchmark-server), these
grow unbounded — confirmed via pprof: 149MB (62% of heap) consumed by
Http2Parser on a pod that OOMs after ~22h.

Cap both payloads at 128KB per stream. They are only used for LLM
provider detection and trace spans which need at most a few KB.
END_STREAM tracking is unaffected so stream completion still works.

* fix: graceful shutdown, L7 panic protection, and bounded strippedGoExeCache (#225)

- Replace os.Exit(0) in signal handler with context cancellation so deferred
  cleanup (cr.Close, profiling.Stop, resolver.StopWatching) runs on shutdown.
  Previously leaked eBPF programs, uprobes, and perf buffers on every restart.
  Use http.Server.Shutdown for graceful HTTP server termination. (fixes #207)

- Wrap L7 event processing with recover() to prevent a single malformed packet
  from crashing the entire agent. Covers both processL7Event and pending L7
  retry paths. (fixes #210)

- Replace unbounded sync.Map with LRU cache (cap 1000) for strippedGoExeCache
  to prevent slow memory growth on CI/CD nodes with many unique Go binaries.
  hashicorp/golang-lru is already a dependency. (fixes #214)

* fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes (#227)

* fix: lightweight HTTP/2 parsing to reduce CPU on busy nodes

HTTP/2 frame parsing was consuming 70% of CPU (profiled). Most HTTP/2
traffic (gRPC, K8s API, service mesh) doesn't need payload capture —
only status codes for L7 metrics.

- Add lightweight mode to Http2Parser: skips DATA payload accumulation
  and RequestHeaders map allocation for non-LLM traffic
- Auto-upgrade to full mode when :authority matches an LLM provider
- Reuse statuses/grpcStatuses maps across Parse() calls (clear instead
  of alloc) — eliminates ~15% CPU from per-call map allocation
- Cap activeRequests at 100 per connection to prevent orphan stream growth
- Cap HTTP/2 parsers at 50 per container
- Fix parser GC leak: connectionless parsers now cleaned up when pid dies
- Eliminate duplicate GetActiveStreamsForLLM() call per event
- Skip LLM stream tracker entirely when parser is lightweight

* fix: address PR review — GC leak for closed connections, deduplicate END_STREAM

- Fix parser GC: closed connection parsers in long-running processes were
  not cleaned up. Now uses heuristic: if pid is alive but no connection
  exists, clean up parsers with zero active requests and no partial data.
- Add HasPartialData() to Http2Parser for safe GC decisions.
- Deduplicate END_STREAM handling: extract it before lightweight/full branch
  instead of duplicating in both paths.

* fix: skip-set for non-LLM HTTP/2 traffic to eliminate CPU waste

processHTTP2WithoutConnection produces zero L7 metrics (no connection =
no DestinationKey). Its only purpose is LLM detection, yet it was parsing
every HTTP/2 frame from gRPC/K8s API traffic — consuming 70% of CPU on
busy nodes.

Two-phase skip-set approach:
1. If dstIP is known, check DNS cache — skip immediately if not LLM.
2. Otherwise, parse first event per pid:fd in lightweight mode to extract
   :authority. If not LLM, add to skip-set and never parse that fd again.

Also adds GC for skip-set entries when the pid dies.

* fix: prevent runtime crash from uint64-to-int overflow in event readers (#228)

* fix: prevent runtime crash from uint64-to-int overflow in event readers

Both perf and ring buffer L7 event readers convert eBPF-reported payload
sizes from uint64 to int without safe clamping. If eBPF sends corrupted
data with high-bit-set sizes (e.g. 0x8000000000000001), int() produces
a negative value that bypasses the MaxPayloadSize cap, causing
make([]byte, negative) to crash the reader goroutine.

Also upgrades cilium/ebpf v0.20.0 → v0.21.0 which includes ring buffer
and memory management fixes relevant to Go 1.24.

Fixes runtime crashes: "fatal error: stack not a power of 2" observed on
12/19 pods after ~7 hours of operation.

* fix: revert cilium/ebpf upgrade, keep clampSize fix only

cilium/ebpf v0.21.0 has breaking API changes (removed Sym, RewriteConstants,
MapName) that affect transitive dependencies. Revert to v0.20.0 and keep
just the uint64-to-int overflow fix.

* fix: evict stale CounterVec label combinations to prevent series explosion (#231)

CounterVec entries for closed TCP connections were never removed,
causing series count to grow linearly with pod uptime. On a 40h pod,
89% of emitted counter series were stale (85/95 destinations in a
single container had no active connection). At customer scale this
accumulated to 989K series (54% of all Prometheus series).

Track label combinations pushed to each CounterVec and periodically
delete entries whose destinations are no longer in activeConnections.
Eviction runs every gcInterval (5min) from the event handler goroutine,
taking mu.Lock to exclude concurrent collect() calls.

* chore: main to test (#237)

* deps: update logparser with structured log level detection (#235)

Updates github.com/nudgebee/logparser to include single-pass
structured log parsing that extracts level from JSON/logfmt fields
instead of unreliable text scanning (nudgebee/logparser#18).

Fixes false positive log level classification where INFO logs
containing "error"/"fatal" in message content were misclassified
as ERROR/CRITICAL.

* chore(deps): bump google.golang.org/grpc from 1.76.0 to 1.79.3 (#229)

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.76.0 to 1.79.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.76.0...v1.79.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.79.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: prevent OOM crash from L7 protocol misidentification (#236)

The ClickHouse eBPF detector matched non-ClickHouse TCP payloads using
a 3-byte heuristic (0x01, 0x00, 0x01), causing ch-go to decode garbage
varint lengths and attempt 228TB allocations — a fatal OOM that bypasses
defer/recover.

3-layer defense:

Layer 0 — eBPF detection hardening:
  Remove non-port-gated is_clickhouse_query() call. ClickHouse native
  protocol detection now only on ports 9000/8123.

Layer 1 — Bounded parsing:
  Wrap ch-go proto.NewReader with io.LimitReader(payload size). Varint
  lengths exceeding actual payload now return io.EOF instead of OOM.

Layer 2 — Userspace validation:
  Add structural checks before invoking external libraries:
  - ClickHouse: validate query code byte + query ID length
  - Postgres: reject unknown frame types (Q/B/P/C only)
  - Zookeeper: validate opcode in known range

Layer 3 — Protocol reclassification:
  Track consecutive parse failures per connection. After 3 failures,
  override the cached protocol to stop further misidentified parsing.
  Logs a warning for observability.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Shiv <3078106+blue4209211@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: graceful shutdown bypasses deferred cleanup due to os.Exit(0)

2 participants