refactor: single-pass structured log parsing for level detection

[mayankpande88]

Summary

• Replace unreliable GuessLevel text-scanning with structured field extraction for JSON/logfmt logs
• ParseStructuredLog() extracts both level and message in a single json.Unmarshal call — eliminates the redundant JSON parse that was happening in the old GuessLevel → NewPattern → normalizeJSONLog pipeline
• Adds logfmt level extraction (level=/lvl=/severity= fields)
• Falls back to existing GuessLevel heuristic for unstructured text logs

Problem

GuessLevel scans raw text for level keywords in the first 255 chars / 7 whitespace-delimited fields. For JSON logs, this causes misclassification:

• INFO log with "type":"ErrorRate" → classified as ERROR
• INFO log with "msg":"error processing request" → classified as ERROR
• Success log with "data":{"observation":"fatal: path already exists"} → classified as CRITICAL

Solution

For structured logs, read the authoritative "level" / "severity" / "levelname" / "log_type" field from the parsed JSON, instead of guessing from raw text. This matches how Datadog, Google Cloud Logging, and OpenTelemetry handle level detection.

JSON level keys supported: level, severity, lvl, log.level, loglevel, log_level, levelname, log_type

Performance (2,575 production logs, Apple M2 Pro)

Path	Per-line	Allocs/line
Old: GuessLevel + NewPattern	9.2µs	61.7
New: ParseStructuredLog + NewPatternFromNormalized	7.7µs	56.8
Result	18% faster	8% fewer allocs

Scale: ~128K lines/sec → 1,276 containers per core at 100 logs/sec each.

Live tested against

• services-server (Go slog, "level" key) — 572 logs, 0 mismatches
• notifications-server (Python, "levelname" key) — 14 logs, 0 mismatches
• forager (Go slog, "level" key) — 28 logs, 0 mismatches
• workspace-server (custom, "log_type" key) — 101 logs, 84 upgraded (unknown→info), 3 false positives fixed

Test plan

• go test ./... passes (all existing + 30 new tests)
• gofmt -l . clean on changed files
• Live tested on 4,607 production logs from dev cluster
• Benchmarked: no performance regression, 18% faster full pipeline
• After deploy: count(container_log_messages_total) by (level) should show only error/critical with fewer false positives

[gemini-code-assist]

Code Review

This pull request introduces structured log parsing for JSON and logfmt formats and optimizes sensitive data detection through anchor-based pre-filtering, shared pattern caching, and entropy-based heuristics. It also adds comprehensive benchmarking and resource impact tests. Feedback focuses on handling escaped quotes in logfmt values, reducing lock contention by moving parsing logic outside critical sections, refining the detection cap to ensure existing patterns are still counted, and optimizing entropy calculations to minimize memory allocations.