chore: main to test#243
Merged
Merged
Conversation
Updates github.com/nudgebee/logparser to include single-pass structured log parsing that extracts level from JSON/logfmt fields instead of unreliable text scanning (nudgebee/logparser#18). Fixes false positive log level classification where INFO logs containing "error"/"fatal" in message content were misclassified as ERROR/CRITICAL.
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.76.0 to 1.79.3. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.76.0...v1.79.3) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.79.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The ClickHouse eBPF detector matched non-ClickHouse TCP payloads using a 3-byte heuristic (0x01, 0x00, 0x01), causing ch-go to decode garbage varint lengths and attempt 228TB allocations — a fatal OOM that bypasses defer/recover. 3-layer defense: Layer 0 — eBPF detection hardening: Remove non-port-gated is_clickhouse_query() call. ClickHouse native protocol detection now only on ports 9000/8123. Layer 1 — Bounded parsing: Wrap ch-go proto.NewReader with io.LimitReader(payload size). Varint lengths exceeding actual payload now return io.EOF instead of OOM. Layer 2 — Userspace validation: Add structural checks before invoking external libraries: - ClickHouse: validate query code byte + query ID length - Postgres: reject unknown frame types (Q/B/P/C only) - Zookeeper: validate opcode in known range Layer 3 — Protocol reclassification: Track consecutive parse failures per connection. After 3 failures, override the cached protocol to stop further misidentified parsing. Logs a warning for observability.
* fix: HTTP2 nil pointer panic and broken GOMEMLIMIT under hostPID
Two fixes found during production health check across 3 clusters:
1. HTTP2 nil pointer dereference (ebpftracer/l7/http2.go):
When maxActiveRequests limit is hit, the break exits the switch but
decoder.Write() still runs with a stale emit func from a previous
call, dereferencing a nil request. Set a no-op emit func before
breaking so HPACK dynamic table stays in sync without the panic.
This was causing 100K+ recovered panics per 6h across KP and DEV,
wasting CPU and inflating page cache by ~150Mi per affected pod.
2. GOMEMLIMIT never set under hostPID (main.go):
readCgroupMemoryLimit() reads /sys/fs/cgroup/memory.max which
doesn't exist under the agent's init.scope cgroup (hostPID places
PID 1 at the host root). The pod's actual 1Gi limit lives at
/sys/fs/cgroup/kubepods.slice/.../cri-containerd-xxx.scope/memory.max.
Fix: read /proc/self/cgroup to resolve the container's own cgroup
path. Without this, GOMEMLIMIT was dead code on all clusters —
GC didn't trigger during ELF parsing spikes, letting RSS hit 1Gi
and causing OOM kills.
* fix: handle cgroup v1 under hostPID in readCgroupMemoryLimit
Parse /proc/self/cgroup for both v2 (hierarchy-id "0") and v1
("memory" controller) to resolve the container's own cgroup path.
Previously only v2 was handled via /proc/self/cgroup; v1 clusters
with hostPID would still fall through to the root path which reports
the host limit, not the pod limit.
fix: aggregate ephemeral workload names to prevent series cardinality explosion
* fix: suppress noisy uprobe "symbol not found" errors for non-matching binaries Binaries like calico-node match the nodejs detection regex but lack uv__io_poll. Similarly, some libpthread.so builds lack pthread_cond_timedwait. These are expected failures, not actionable errors — suppress them like other benign attachment failures. * fix: narrow suppression to "symbol ... not found" errors only Generic "not found" suffix could mask unrelated errors. Match the specific error format from elf.go GetSymbol instead.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces workload aggregation for ephemeral Kubernetes pods and standalone Jobs to reduce metric cardinality by using standard labels or a namespace-based fallback. It also enhances L7 protocol detection and parsing robustness, specifically adding structural validation for ClickHouse, Postgres, and ZooKeeper to prevent OOMs and misidentification issues. Additionally, it improves cgroup memory limit detection for agents running with hostPID by parsing /proc/self/cgroup and updates several project dependencies. I have no feedback to provide.
RamanKharchee
approved these changes
Apr 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.