Security ML rule: use event.ingested for deduplication-safe anomaly queries #259297
Description
Summary
Update the Security ML detection rule to query .ml-anomalies-* by event.ingested (result index write time) instead of timestamp (bucket time). This eliminates duplicate alerts caused by the same anomaly document matching across multiple consecutive rule runs.
Elasticsearch dependency: elastic/elasticsearch#144836 adds event.ingested to ML result documents. Kibana issue: #259287.
Background
The ML rule executor (ml.ts → find_ml_signals.ts → buildAnomalyQuery in server/lib/machine_learning/index.ts) filters anomaly records with a range.timestamp window derived from the rule interval. The same document can match on every run until the window advances past it, generating duplicate detections.
Once elastic/elasticsearch#144836 lands, each result document carries event.ingested — the exact index write time. Filtering by event.ingested >= last_run_time ensures each document is matched exactly once.
What needs to change
File: x-pack/solutions/security/plugins/security_solution/server/lib/machine_learning/index.ts
Function: buildAnomalyQuery / buildCriteria
Add a fallback dual-range query (see #259287 for the pattern): for documents with event.ingested, use event.ingested >= last_run_time; for documents without it (legacy), keep the existing timestamp range. This handles indices in transition and requires no feature flag.
Callers in find_ml_signals.ts and ml.ts may need minor plumbing to pass last_run_time alongside the existing earliestMs/latestMs.
Related
- Add
event.ingestedto ML anomaly detection result documents elasticsearch#144836 — addsevent.ingestedto.ml-anomalies-*documents - [ML] Anomaly detection alerting: use
event.ingestedfor deduplication-safe rule queries #259287 — Kibana ML alerting issue (includes the dual-range query pattern)