fix: optimize sensitive log parsing performance and reduce false positives

[mayankpande88]

Summary

• Add anchor-based pre-filtering that skips ~91% of regex evaluations on normal log lines (21-55x faster)
• Add confidence tiers (high/medium/low) to all 200 sensitive patterns, with configurable minimum confidence level
• Add Shannon entropy + character-class validation to reject false positives on low-confidence patterns (e.g. SQL table names, cache keys)
• Add singleton pattern cache to share compiled regexes across parsers (~14KB vs ~2.1MB per parser)
• Replace bare bool parameter with SensitiveConfig struct supporting sampling rate, min confidence, and max detections cap
• Add configurable 1-in-N line sampling and per-container detection cap

Test plan

• go test ./... passes
• TestExtractAnchors validates anchor extraction from regex patterns
• TestPrefilterEffectiveness confirms ≥80% anchor coverage
• TestPrefilterSkipsNonMatching confirms ≥90% skip rate on normal lines
• TestLooksLikeSecret validates entropy-based FP rejection
• TestDetectSensitiveData_LowConfidence_EntropyFilter confirms real secrets detected, SQL table names rejected
• TestParserSensitiveDisabled confirms zero overhead when disabled
• TestParserSensitiveSampling and TestParserSensitiveMaxDetections validate rate limiting
• Benchmark: go test -bench=. -benchmem shows improvement over baseline

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive set of enhancements to the sensitive log parsing mechanism. The primary goal is to drastically improve performance and accuracy by intelligently filtering log lines before full regex evaluation and by applying more robust validation to potential matches. These changes provide greater control over the detection process and ensure that the system is both efficient and reliable in identifying sensitive data.

Highlights

• Performance Optimization: Implemented anchor-based pre-filtering that significantly reduces regex evaluations on normal log lines, leading to 21-55x faster processing.
• False Positive Reduction: Introduced Shannon entropy and character-class validation to reject false positives, especially for low-confidence patterns like SQL table names or cache keys.
• Configurable Confidence Tiers: Assigned confidence tiers (high, medium, low) to all 200 sensitive patterns, allowing for configurable minimum confidence levels.
• Memory Efficiency: Added a singleton pattern cache to share compiled regexes across parsers, reducing memory footprint from ~2.1MB to ~14KB per parser.
• Enhanced Configuration: Replaced a bare boolean parameter with a SensitiveConfig struct, enabling granular control over sampling rate, minimum confidence, and maximum detections per container.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• bench_loki_test.go
  • Added a new benchmark file to test sensitive data detection performance against real logs fetched from Loki.
• bench_test.go
  • Added new benchmarks to measure the performance of sensitive data detection, including comparisons with the old approach and pre-filter-only scenarios.
• cmd/logparser.go
  • Updated the NewParser instantiation to utilize the new SensitiveConfig struct for sensitive data detection.
• impact_test.go
  • Added a new test file to measure the CPU and memory resource impact of sensitive data detection under various configurations.
• parser.go
  • Added strings package import.
  • Introduced a global patternCache with mutex for sharing compiled regexes across parsers.
  • Modified PrecompiledPattern struct to include Anchors for pre-filtering and Confidence level.
  • Defined SensitiveConfig struct to encapsulate sensitive data detection parameters.
  • Updated Parser struct to store SensitiveConfig and a sensitiveCounter for sampling.
  • Refactored NewParser to accept SensitiveConfig and load patterns using the new caching mechanism.
  • Converted processSensitivePattern to a method of Parser and integrated sampling and max detections logic.
  • Updated SensitivePattern struct to include a Confidence field for JSON unmarshalling.
  • Added confidenceLevel helper function to map confidence strings to numeric levels.
  • Modified DetectSensitiveData to incorporate anchor-based pre-filtering and looksLikeSecret post-validation for low-confidence patterns.
  • Implemented getOrLoadPatterns to retrieve patterns from the shared cache or load them if not present.
  • Updated LoadPatterns to filter patterns based on minConfidence and to extract anchors for each pattern.
• parser_test.go
  • Updated TestParser to initialize the parser with SensitiveConfig.
  • Added TestParserSensitiveDisabled to verify that sensitive detection has no overhead when disabled.
  • Added TestParserSensitiveSampling to test the 1-in-N line sampling functionality.
  • Added TestParserSensitiveMaxDetections to ensure the detection cap works correctly.
  • Added TestParserMinConfidence to validate the filtering of patterns by confidence level.
  • Initialized sensitivePatterns in TestParserCardinalityLimit for consistency.
• sensitive_filter.go
  • Added a new file containing utility functions for sensitive data filtering.
  • Implemented extractAnchors to derive literal substrings from regexes for pre-filtering.
  • Added leadingLiteral to extract literal prefixes from strings.
  • Created anchorMatchesLine to efficiently check for anchor presence in a log line.
  • Developed shannonEntropy to calculate the entropy of a string.
  • Implemented looksLikeSecret to validate potential sensitive matches using entropy and character-class analysis.
  • Added dedupStrings helper function.
• sensitive_filter_test.go
  • Added a new test file for the sensitive filtering logic.
  • Included TestExtractAnchors to validate anchor extraction from various regex patterns.
  • Added TestAnchorMatchesLine to test the anchor matching functionality.
  • Implemented TestPrefilterEffectiveness to verify the coverage of anchor extraction across patterns.
  • Added TestPrefilterSkipsNonMatching to confirm the pre-filter's ability to skip irrelevant lines.
  • Included TestLooksLikeSecret to test the false positive rejection logic based on entropy and character classes.
  • Added TestShannonEntropy to verify entropy calculation.
  • Implemented TestDetectSensitiveData_LowConfidence_EntropyFilter to confirm correct detection and false positive rejection for low-confidence patterns.
  • Added TestDetectSensitiveDataWithPrefilter to test sensitive data detection with pre-filtering enabled.

Activity

• The author mayankpande88 initiated this pull request to address performance and accuracy issues in sensitive log parsing.
• The pull request includes a detailed summary of the changes and a comprehensive test plan, indicating thorough consideration of the new features.
• New benchmark and impact tests have been added to validate the performance improvements and resource usage of the updated detection mechanisms.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces significant performance optimizations and accuracy improvements for sensitive data detection in logs. The changes include anchor-based pre-filtering, confidence levels for patterns, entropy-based validation, and a shared cache for compiled regexes. The implementation is solid and backed by extensive new benchmarks and tests. I've found a few areas for improvement, mainly related to test robustness and minor code cleanup, which I've detailed in the comments.