Updating fields to new ECS names

[ycombinator]

Motivated by https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519.

This PR updates the documentation on https://www.elastic.co/guide/en/logstash/7.6/logstash-config-for-filebeat-modules.html. Specifically, it updates the sample Logstash pipelines shown on that page to reference ECS field names added by Beats modules instead of of old-style names.

[dedemorton]

Looks like this PR addresses the issue here: #10490

@karenzone I'm assuming that you want to review this because I know you had this on your pile. Let me know if you need anything from me. Also, I think this PR is old and needs closing, right? #11563

[webmat]

Changing these two fields is a good start.

But having a cursory look at the rest of the pipelines, it looks like they're trying to reproduce the Beats module's parsing of the logs. Each of them has changed radically since 7.0, so the rest of these pipelines will not yield the expected equivalence. Just compare with the groks over in Beats (e.g. Apache HTTPD pipeline)

Note also that the "apache2" module has been renamed to "apache" in Beats.

[ycombinator]

Thanks @webmat, good catch on the rest of the pipeline and the module name! I will update this PR to use the new module name and also try to remove out any redundant bits of parsing being done in the LS pipeline that's already being handled by the Beat module. I believe the intent of this doc is to show how to further enrich data coming from Beats in a Logstash pipeline, e.g. with the geoip filter plugin, so I'll try to keep the LS pipeline contents down to just such enrichment(s).

[webmat]

Makes a ton of sense. You'll want to pay special attention to how each module is built, though.

Beats has been adding lots of processors recently, and the most recent modules have moved some of the processing to Beats processors, I think that a lot of older modules are still doing this entirely in Elasticsearch pipelines.

So in the latter cases, Logstash will be getting essentially the raw unparsed Filebeat event. These could be used to demonstrate how to duplicate & replace the functionality of these pipelines.

In the cases where significant processing has happened in Beats already (e.g. look at the Zeek and Suricata modules, for example), then you'll be able to use Logstash to do supplemental work on the logs without duplication. Note that even these modules often have further things happening in ES ingest node, like geoip & ASN lookups.

[dedemorton]

This PR seems to be stalled.

[karenzone]

This work is still on hold pending larger ECS strategy for Logstash/ECS

[kares]

LGTM, these are worth revisiting/updating with ECS being on by default in 8.x