Save a summarize and use correct Time column in most TI Map Detections#3477
Save a summarize and use correct Time column in most TI Map Detections#3477ep3p wants to merge 65 commits into
Conversation
|
Some Detection Rules do not have a 2nd summarize from before this pull request, in case you want them all to have it. |
Fixing up mapping for IP.Address. Specifing join kind explicitly and commenting as to why we use innerunique. Making changes to filtering after the join and fixing the 2nd argmax to use the activity time instead of indicator time as you want the most recent activity that matches, both changes per discussions on another PR #3477.
Removed account for now, not sure there is a mapping for account, Ajeet to look at when he is back. Fixed indent on entitymapping. Moved | where TimeGenerated >= ago(dt_lookBack) up in the AzureDiagnostic query as time is the primary index. Specifying join kind explicitly and commenting as to why we use innerunique. Making changes to filtering after the join and fixing the 2nd argmax to use the activity time instead of indicator time as you want the most recent activity that matches, both changes per discussions on another PR #3477.
Per our other PR Azure#3477, making the same changes to the 2nd summarize which should be on the activity table time, in this case SigninLogs and not TI lastindicatetime
|
Hi @shainw , I have checked the next code exists in all TI map detections: ThreatIntelligenceIndicator
| where TimeGenerated >= ago(ioc_lookBack) and ExpirationDateTime > now()if Expiration is checked at first, I don't think it makes much sense to check it again later. on $left.EmailSenderAddress == $right.Caller
| where AzureActivity_TimeGenerated < ExpirationDateTimeDo you mind if I remove the last part after the join? Moreover, out of curiosity. please, when comparing to SecurityAlerts, I don't understand why doing: | extend MSTI = case(AlertName has "TI map" and VendorName == "Microsoft" and ProductName == 'Azure Sentinel', true, false)
| where MSTI == falseinstead of | extend not(AlertName has "TI map" and VendorName == "Microsoft" and ProductName == 'Azure Sentinel) |
|
I am going to close this Pull Request, and open 3 different ones with the changes I wanted to apply here. |
|
@shainw Not sure if in the right area for this, but I'm noticing a lot of the TI indicator detection rules doesn't seem to take into account the ingestion time. (Reference: https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/handling-ingestion-delay-in-azure-sentinel-scheduled-alert-rules/ba-p/2052851) If by example, we use this rule (https://github.com/Azure/Azure-Sentinel/blob/master/Detections/ThreatIntelligenceIndicator/URLEntity_AuditLogs.yaml) if i verify delay's between ingested time VS TimeGenerated within my AuditLogs table, i sometime get 44 minutes of delays. Some tables have bigger delays then others, but if the delay is higher than the "dt_lookback" value this could result in missed alerts.
|
|
@PYB-Machine I think there is one TI map rule that uses ingestion time (https://github.com/Azure/Azure-Sentinel/blob/master/Detections/ThreatIntelligenceIndicator/DomainEntity_CommonSecurityLog.yaml), in case you want to apply the same to the rest. |
3 participants
Fixes #
TimeGenerated comes from the table you want to check, NOT from ThreatIntelligenceIndicator (its TimeGenerated was substituted by LatestIndicatorTime).
It does not make sense to compare "Source_TimeGenerated >= TimeGenerated", when they are the same value. "Source_TimeGenerated" should be compared to "LatestIndicatorTime".
"extend Source_TimeGenerated = TimeGenerated" should not have been used, but "project-rename Source_TimeGenerated = TimeGenerated".
Therefore a second summarize computing LatestIndicatorTime is not needed, multiple events related to the same Indicator are lost in the results.
More changes might be needed, depending on the results.
Please, could you check this?
Proposed Changes