[Security Solution] Gracefully handle `data_stream.*` fields in alert source data

[ymao1]

The alerting framework considers the data_stream.* fields to be reserved fields in the alerts as data indices, particularly since AAD in serverless are actually datastreams so we may want to set specific values for these fields in the future.

Currently, data_stream.* fields from the source index are copied over as-is into data_stream.* fields in the alert document. This was causing issues with the default ECS mapping for these fields (which is constant_keyword) so these fields are actually unmapped in the alert document.

To handle these fields more gracefully, we suggest either
(1) Copying data_stream.* fields into a different field within the alerts index that security solutions can define a mapping for (similar to how event.* fields are copied into kibana.alert.original_event.*).
(2) Stripping these fields altogether when building the alert document. It seems that event.dataset === data_stream.dataset so that information is already captured in the source document in a different field anyway.

Open to other solutions but the end result should be that the alert document built by the detection engine does not contain the data_stream.* fields in their original path.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[NeilADesai]

This causes issues when trying to show the power of Elastic and removing a lot of alerts that are not needed to be seen for an investigation or a threat hunt.
In the EDEN AWS demo there are a lot of alerts from 'data_stream.dataset : cloud_security_posture.findings' that should be able to be filtered out instead of having to do each alert by name.

[ar3diu]

I'm also interested in this issue. I would like to be able to use data_stream.namespace field in exceptions and alert searches.

[ar3diu]

Any updates on this? :)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[rylnd]

In the EDEN AWS demo there are a lot of alerts from 'data_stream.dataset : cloud_security_posture.findings' that should be able to be filtered out instead of having to do each alert by name.

@NeilADesai did you see @ymao1's comment about event.dataset? Both that field and kibana.alert.original_event.dataset should contain the same value as data_stream.dataset, but more importantly the former fields are already mapped in our alerts indices, so should be suitable for your filtering purposes.

[rylnd]

Disclaimer, first: it's currently possible to accomplish this functionality with a runtime field for filtering on the alerts dataview (e.g. original_namespace: emit(params['_source']['data_stream']['namespace'])).

Now, on to the discussion about having this field automatically copied/filterable:

@ymao1 given the alerting framework's stance on data_stream.* I would agree that we should not be copying those values wholesale to the alert document. That should be relatively straightforward, as we already have a blocklist mechanism.

Since we have no equivalent for e.g. data_stream.namespace, I think it makes sense to copy that (and possibly a few more fields) to e.g. kibana.alert.original_data_stream.*. I'm not sure what the current process looks like for defining/adding mappings for these kibana.alert.* fields, but I think the addition of a few for this use case is reasonable here.

@marshallmain: what do you think about this, and about the destination being kibana.alert.original_data_stream.* ?

@NeilADesai @ar3diu : what fields would be most useful to have for this purpose (and please distinguish between needing the value on the document vs. needing it mapped/searchable).

[ar3diu]

what fields would be most useful to have for this purpose (and please distinguish between needing the value on the document vs. needing it mapped/searchable).

In a multi-tenant/MSSP scenario, most of the users will use the namespace field to separate log data from customers in different data streams.
Therefore, it would be useful to be able to search for documents with specific data_stream.namespace values in the Kibana alert indices.