chore: main to test

[mayankpande88]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @mayankpande88, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement to the agent's observability capabilities by adding comprehensive support for Large Language Model (LLM) API tracing and metrics. It enables detailed monitoring of LLM interactions, including streaming responses, by leveraging advanced eBPF techniques for Go TLS and HTTP/2 traffic. The changes ensure more reliable data collection and provide deeper insights into LLM performance and usage patterns.

Highlights

• LLM Observability: Introduced comprehensive tracing and metrics for Large Language Model (LLM) API interactions, including support for streaming responses (Server-Sent Events) from providers like OpenAI, Anthropic, Google Gemini, and AWS Bedrock.
• Dynamic Go TLS Tracing: Implemented dynamic discovery of Go TLS internal offsets using DWARF information or version-based fallbacks, significantly improving the reliability and compatibility of Go TLS tracing across different Go versions and binaries, including gRPC syscallConn wrappers.
• Enhanced eBPF L7 Event Handling: Upgraded the eBPF L7 tracer to use a ring buffer for event processing, ensuring global ordering crucial for streaming data. It also gains the ability to extract socket information directly from file descriptors, mitigating Go TLS connection tracking issues.
• Detailed LLM Metrics: Added new Prometheus metrics for LLM requests, token usage (input/output), time to first token (TTFT), request duration, tokens per second, and categorized errors, providing granular insights into LLM API performance and usage.
• Robust LLM Provider Detection: Improved LLM provider detection logic to consider hostname, URL path patterns, and request/response payload structure, enhancing accuracy for various LLM services, including gRPC-based ones.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• containers/container.go
  • Removed unused 'log' import.
  • Added 'Operation' field to 'LLMStats' struct for OTel GenAI operation names.
  • Introduced 'googleHTTP2Parsers' map and 'llmStreamTracker' to the 'Container' struct.
  • Refactored 'NewContainer' to differentiate workload resolution for Kubernetes and non-Kubernetes containers.
  • Initialized and stopped 'llmStreamTracker' in 'NewContainer' and 'Close' methods respectively.
  • Optimized 'Collect' method by removing verbose logging and adding read locks for concurrent map access.
  • Added 'createConnectionFromSocketInfo' to create active connections from eBPF socket info, improving Go TLS tracking.
  • Updated 'trackLLMRequest' to include path, use 'GetOperation', and emit Prometheus metrics directly.
  • Implemented 'onLLMStreamComplete' to record Prometheus metrics and emit OTel traces for completed LLM streams.
  • Modified 'onL7Request' to 'onL7RequestWithResult' to handle connection tracking failures and HTTP/2 specific processing.
  • Added 'processHTTP2WithoutConnection' for handling HTTP/2 events when TCP connection tracking is unavailable.
  • Refactored 'updateDelays' to reduce lock contention during PID snapshotting and syscalls.
  • Removed debug logging for SSL uprobes.
  • Updated LLM detection logic to use centralized path-based and payload-structure-based detection.
• containers/http_processor.go
  • Added 'bytes' and 'netaddr' imports.
  • Introduced 'DNSResolver' type for dynamic hostname resolution.
  • Enhanced 'HTTPRequestContext' with SSE detection and LLM provider caching.
  • Renamed 'NewHTTPRequestProcessor' to 'NewHTTPRequestContext' and updated its signature.
  • Refactored request parsing, host resolution, payload encoding, and trace ID extraction for clarity and robustness.
  • Added helper functions for stripping ports, extracting HTTP body/headers, and parsing W3C Trace IDs.
• containers/llm.go
  • Added 'net', 'net/url', and 'regexp' imports.
  • Updated 'LLMProvider' enum names to align with OTel standards (e.g., 'gcp.gemini').
  • Defined OTel GenAI operation name constants.
  • Added 'Operation' field to 'LLMRequest' struct.
  • Expanded 'llmProviders' map and added regex patterns for AWS Bedrock and Azure OpenAI hostnames.
  • Introduced 'providerDefaultHost' function for canonical hostnames.
  • Improved 'DetectLLMProvider' to handle port stripping and regex-based hostname matching.
  • Added 'DetectLLMProviderFromPath' for path-based LLM provider and model detection.
  • Implemented helper functions for extracting Google Gemini and AWS Bedrock models from paths.
  • Added 'GetOperation' to derive OTel GenAI operation from request path.
  • Updated 'ParseLLMRequest' and 'ParseLLMResponse' to include path, use 'extractJSON', and support AWS Bedrock parsing.
  • Added 'extractJSON' helper and provider-specific parsing functions for AWS Bedrock.
• containers/llm_grpc.go
  • Added new file to provide gRPC-specific LLM helpers.
  • Implemented 'ExtractGRPCServiceMethod' to parse gRPC paths.
  • Added 'IsGeminiGRPCService' to identify Google Gemini gRPC services.
• containers/llm_metrics.go
  • Added new file to define Prometheus metrics for LLM observability.
  • Defined 'ContainerLLMRequestsTotal', 'ContainerLLMTokenUsageTotal', 'ContainerLLMTimeToFirstToken', 'ContainerLLMRequestDuration', 'ContainerLLMTokensPerSecond', and 'ContainerLLMErrorsTotal' as Prometheus vectors.
  • Provided 'RegisterLLMMetrics' function to register all LLM-related metrics.
  • Implemented 'RecordLLMStreamMetrics' to populate these metrics from LLM stream data.
  • Added 'categorizeHTTPError' helper for mapping HTTP status codes to error types.
• containers/llm_stream.go
  • Added new file to manage LLM streaming requests.
  • Defined 'LLMStream' struct to track individual LLM streams with detailed metadata.
  • Implemented 'LLMStreamTracker' to manage the lifecycle of multiple LLM streams.
  • Provided methods for handling request headers, response headers, data frames, and stream ends.
  • Included logic for detecting SSE completion markers and extracting token counts from stream buffers.
  • Implemented garbage collection for idle or long-running streams.
  • Added helper functions for trace ID generation and provider-specific token extraction.
• containers/registry.go
  • Added 'pendingL7Events' queue and lock to handle out-of-order L7 events.
  • Registered LLM metrics in 'NewRegistry'.
  • Removed verbose Prometheus registration/unregistration logging.
  • Modified 'handleEvents' to process L7 events and retry pending ones.
  • Implemented 'processL7Event' and 'queueL7EventForRetry' for robust L7 event handling.
  • Added 'processPendingL7Events' to manage retries of L7 events.
  • Included 'ignoreControlPlane' check for systemd services in 'calcId'.
• ebpftracer/ebpf/ebpf.c
  • Added conditional compilation for 'bpf_printk' using 'BPF_DEBUG' macro.
• ebpftracer/ebpf/l7/gotls.c
  • Introduced 'go_tls_offsets' struct and map for dynamic Go TLS offset discovery.
  • Refactored 'go_crypto_tls_get_fd_from_conn' to use dynamic offsets and handle gRPC 'syscallConn' wrappers.
  • Added 'ensure_connection_tracked' to address Go goroutine thread-switching issues in TCP tracking.
  • Modified 'go_crypto_tls_write_enter' and 'go_crypto_tls_read_enter' to call 'ensure_connection_tracked' and include debug logging.
• ebpftracer/ebpf/l7/http2.c
  • Updated HTTP/2 frame type and flag definitions.
  • Refactored HTTP/2 client preface detection.
  • Added 'parse_http2_frame_header' for robust frame header validation.
  • Implemented 'detect_hpack_static_index' for HPACK static table index detection.
  • Added 'is_headers_frame_with_hpack' for stricter HEADERS frame validation.
  • Refactored 'looks_like_http2_frame' for improved HTTP/2 detection logic, including multi-frame scanning.
  • Added 'is_likely_http2_port' for port-based HTTP/2 hints.
• ebpftracer/ebpf/l7/l7.c
  • Added 'COPY_PAYLOAD_RINGBUF' macro for ring buffer operations.
  • Included 'socket_info.c' for socket information extraction.
  • Extended 'l7_event' struct with socket tuple fields.
  • Replaced perf event array with a ring buffer for 'l7_events'.
  • Introduced 'send_event', 'reserve_l7_event', and 'discard_l7_event' for ring buffer management.
  • Modified 'send_event' to include socket information.
  • Updated 'trace_enter_write' and 'trace_exit_read' with fast paths for HTTP/2, socket info fallbacks, and ring buffer usage.
• ebpftracer/ebpf/l7/openssl.c
  • Modified 'WRITE_ENTER' and 'READ_ENTER' macros to call 'ensure_connection_tracked' for OpenSSL TLS.
• ebpftracer/ebpf/socket_info.c
  • Added new file for kernel socket information extraction via BTF.
  • Defined 'socket_info_offsets' struct and map for dynamic offset discovery.
  • Implemented 'get_socket_tuple_from_fd' to extract socket details from a file descriptor.
• ebpftracer/ebpf/tcp/state.c
  • Increased 'MAX_PAYLOAD_SIZE' to 8192 bytes for larger LLM HTTP/2 payloads.
• ebpftracer/go_offsets.go
  • Added new file for Go TLS offset discovery.
  • Defined 'GoTLSOffsets' and 'GoTLSOffsetsC' structs.
  • Included 'knownGoOffsets' for version-based fallbacks.
  • Implemented 'DiscoverGoTLSOffsets' using DWARF and fallbacks.
  • Added 'DiscoverItabAddresses' to find itab addresses for gRPC 'syscallConn' wrappers.
• ebpftracer/l7/clickhouse.go
  • Added a 'recover' block to 'ParseClickhouse' to handle panics from malformed packets.
• ebpftracer/l7/http2.go
  • Added 'klog' import and HTTP/2 flag constants.
  • Extended 'Http2Request' with new fields for Authority, ContentType, RequestHeaders, and internal state for stream tracking.
  • Introduced 'pendingHeaderBlock' for header reassembly.
  • Added partial frame and pending header buffers to 'Http2Parser'.
  • Implemented 'resetDecoder' for HPACK error recovery.
  • Added 'ActiveRequestCount' and 'GetActiveStreamsForLLM' for stream management.
  • Refactored 'Parse' method to handle partial frames, process various HTTP/2 frame types, and manage HPACK decoding.
• ebpftracer/socket_offsets.go
  • Added new file for managing kernel socket offsets.
  • Defined 'SocketInfoOffsets' struct and 'discoverSocketOffsets' using BTF.
  • Implemented 'initSocketInfoOffsets' to populate the eBPF map.
  • Added 'SocketInfo' struct and 'GetSocketInfoFromL7Event' for user-space socket info extraction.
• ebpftracer/tls.go
  • Added 'encoding/binary' and 'unsafe' imports.
  • Expanded TLS symbol detection to include S2A and additional HTTP/2 related functions.
  • Updated regex patterns for 'libssl' and 'libcrypto' for broader matching.
  • Enhanced 'AttachGoTlsUprobes' with more debug logging and calls to 'populateGoTLSOffsets'.
  • Implemented 'populateGoTLSOffsets' to write discovered Go TLS offsets to the eBPF map.
• ebpftracer/tracer.go
  • Added 'log', 'sync', and 'sync/atomic' imports.
  • Increased 'MaxPayloadSize' to 8192 bytes.
  • Removed 'EventTypeHTTPFragment' and 'HTTPFragment' from 'Event' struct.
  • Added 'SocketInfo' to 'Event' struct.
  • Introduced 'ringbufReader' to 'Tracer' struct and ensured its closure.
  • Initialized socket info offsets in 'ebpf' function.
  • Migrated L7 event reading from perf buffer to ring buffer using 'runRingbufEventsReader'.
  • Implemented 'lostSamplesTracker' for improved logging of lost eBPF samples.
  • Refactored 'runEventsReader' for performance and to use 'lostSamplesTracker'.
  • Added 'runRingbufEventsReader' to process L7 events from the ring buffer, including socket info extraction.
  • Modified 'ipPort' to accept a byte slice for IP addresses.
• go.mod
  • Updated various Go module dependencies, including OpenTelemetry, Google Cloud SDK, and testing libraries.
  • Added new dependencies for path handling.
• go.sum
  • Updated checksums for Go module dependencies.
• main.go
  • Added 'flag' import.
  • Modified 'main' function to properly initialize 'klog' flags and redirect output to a rate-limited logger.
• proc/net.go
  • Removed 'net' import.
  • Refactored 'decodeAddr' to use stack-allocated buffers and optimize IP address decoding for IPv4 and IPv6 from /proc/net/tcp and /proc/net/udp.
• tracing/tracing.go
  • Added 'LLMStreamInfo' struct to encapsulate LLM stream details for tracing.
  • Implemented 'LLMRequest' method to create OpenTelemetry spans for LLM API requests, including detailed attributes for provider, model, operation, token usage, TTFT, and error handling.

Activity

• The pull request was created by mayankpande88.
• The pull request modifies several core components related to eBPF tracing, container management, and LLM integration.
• New files were added to support LLM gRPC parsing, LLM metrics, LLM stream tracking, and kernel socket offset discovery.
• Existing eBPF programs and Go code were updated to integrate these new features and improve robustness.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a substantial set of enhancements focused on improving LLM and gRPC observability, along with significant architectural and performance improvements. The changes are extensive and well-executed, greatly increasing the agent's capabilities and robustness.

Key improvements include:

• Enhanced LLM & gRPC Tracing: A new streaming LLM tracker has been added to monitor responses using Server-Sent Events (SSE), and the agent now adheres to OpenTelemetry's GenAI semantic conventions for metrics and traces.
• Robust Go TLS Probing: The mechanism for tracing Go applications has been completely overhauled. It now uses dynamic offset discovery (via DWARF, itab, and BTF) to make probes resilient to different Go and kernel versions. Fallback paths have been added in eBPF to handle cases where standard TCP connection tracking fails, which is common in Go applications using goroutines.
• eBPF Modernization: The L7 event handling has been migrated from perf buffers to a more modern ring buffer, which guarantees event ordering and improves performance. The HTTP/2 frame parsing logic in eBPF has also been made significantly more compliant and strict to prevent false positives.
• Concurrency and Performance: Several potential race conditions in the metrics collection code have been resolved by implementing proper locking. Additionally, performance has been improved in hot paths by reducing lock contention and eliminating heap allocations.

Overall, this is an excellent update that demonstrates a deep understanding of eBPF and observability challenges in modern applications. The code is well-refactored and the new features are thoughtfully implemented.