fix: stable JSON log pattern hashing using message-field extraction#16
Merged
Conversation
normalizeJSONLog was dumping all JSON field values into the pattern, causing empty hashes when large data blobs (HTML, stack traces) overwhelmed the extractor, and unstable hashes when metadata fields (timestamps, trace IDs, file paths) changed between log entries. Now extracts patterns only from recognized message fields (msg, message, error, err, reason, log, text) with case-insensitive key matching, following industry standard (Datadog, New Relic, Elastic). Falls back to all values with a per-field length cap when no message key is found. Also fixes TestParser: test data didn't match any sensitive pattern regex (literal "Key" vs required 10+ char credential).
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
RamanKharchee
approved these changes
Mar 10, 2026
This was referenced Mar 10, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
normalizeJSONLog: Extract patterns only from recognized message fields (msg,message,error,err,reason,log,text) with case-insensitive key matching, instead of dumping all JSON values. Falls back to all values with a 200-char per-field cap when no message key is found.TestParser: Fix test data — literal"Key"doesn't match any sensitive pattern regex (requires 10+ char credential). Now usesAKIAIOSFODNN7EXAMPLE. Sleep reduced from 10s to 3s.Problem
d41d8cd9...(MD5 of empty string)TestParseralways failing: Test credential"Key"was too short to match any sensitive pattern regexTest plan
TestJsonPatternRealLokiLogs— 13 hand-picked production logs (Go zerolog, Python, custom JSON)TestJsonPatternBulkLokiLogs— 459 unique production logs from Loki, 0 empty-hash violationsTestJsonPatternMessageFieldOnly— hash stability across metadata changesTestJsonPatternCaseInsensitiveKeys—Message,message,MSGall workTestJsonPatternFallback— no-message-key path still worksTestJsonPatternWithErrorField— error field differentiationTestParser— now passes (was permanently broken before)