chore: fixes for llm tracking

[mayankpande88]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @mayankpande88, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant enhancements to the system's ability to monitor and trace LLM (Large Language Model) interactions and other L7 traffic. It focuses on improving the reliability and accuracy of data collection by addressing common challenges in eBPF-based tracing, such as race conditions in connection tracking and dynamic changes in Go runtime and kernel structures. The changes ensure that L7 metrics are more compliant with OpenTelemetry standards and that tracing works effectively across diverse application and kernel environments.

Highlights

• Enhanced LLM Tracking: LLM tracking has been significantly improved by aligning with OpenTelemetry GenAI semantic conventions. This includes adding an 'Operation' field to LLMStats, updating provider names (e.g., 'gcp.gemini', 'aws.bedrock'), and introducing new metrics ('gen_ai_client_token_usage', 'gen_ai_client_operation_duration_seconds').
• Robust L7 Event Processing: The L7 event processing pipeline has been made more robust, particularly for Go TLS and HTTP/2 traffic. This includes a new mechanism to queue and retry L7 events that arrive before their corresponding TCP connections are fully established, addressing race conditions.
• Dynamic Offset Discovery for Go TLS: The eBPF tracer now dynamically discovers Go TLS offsets using DWARF debug information or version-based fallbacks. This makes the tracer more resilient to changes in Go compiler versions and improves the accuracy of FD extraction from Go TLS connections, including support for gRPC's syscallConn wrapper.
• Direct Socket Info Extraction in eBPF: eBPF programs can now directly extract socket tuple information (source/destination IP/port, family) from file descriptors using BTF-discovered kernel offsets. This allows L7 events to be processed even when traditional TCP connection tracking fails, a common issue with Go's goroutine scheduling.
• Improved HTTP/2 Detection and Parsing: HTTP/2 frame detection and parsing logic in eBPF has been refined with better client preface detection, frame header validation, and HPACK static table index analysis. A new 'connectionless' HTTP/2 processing path handles events for untracked connections.
• Transition to Ring Buffer for L7 Events: L7 events are now processed using an eBPF ring buffer instead of per-CPU perf buffers. This provides global event ordering, which is crucial for correctly processing streaming responses like Server-Sent Events (SSE).
• Increased Payload Size for L7 Tracing: The maximum payload size for L7 tracing has been increased to 8192 bytes to better accommodate larger request/response bodies, especially for LLM API calls.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces significant enhancements to LLM tracking, HTTP/2 parsing, and the overall robustness of eBPF tracing, especially for Go TLS applications. Key improvements include aligning LLM metrics with OpenTelemetry GenAI semantic conventions, implementing dynamic Go TLS offset discovery, and introducing a connectionless HTTP/2 processing mechanism. The use of a ring buffer for L7 events and a retry mechanism for pending events greatly improves reliability and handles race conditions. Additionally, several map iterations are now protected by read locks to prevent concurrent access issues.