Skip to content

parser: clarify that multiline parser does not join separate events#649

Merged
kenhys merged 2 commits into
fluent:1.0from
Watson1978:clarify-multiline-parser-scope
Jul 24, 2026
Merged

parser: clarify that multiline parser does not join separate events#649
kenhys merged 2 commits into
fluent:1.0from
Watson1978:clarify-multiline-parser-scope

Conversation

@Watson1978

@Watson1978 Watson1978 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Closes: #338

MultilineParser#parse in lib/fluent/plugin/parser_multiline.rb applies the regexp built from formatN (compiled with Regexp::MULTILINE) to the text it is given, then loops over Regexp::MatchData#post_match and yields once per match. So a single call can emit several records, but every bit of state is local to that call: the plugin keeps no buffer across calls, and therefore cannot combine lines which arrive as separate events. It only exposes has_firstline? / firstline? as helpers for the caller.

The actual line buffering is in lib/fluent/plugin/in_tail.rb: it detects the multiline mode with @multiline_mode = parser_config["@type"].include?("multiline"), and parse_multilines appends lines to a buffer until @parser.firstline?(line) matches the next first line.

lib/fluent/plugin/filter_parser.rb has no guard against multiline: no ConfigError, no warning. So a <filter> with @type parser and @type multiline in its <parse> section silently parses a single field of an already-split event, which is not what users expect. That is the confusion reported in #338. On top of that, filter_one_record adds a new event per yield, so one input event can even be expanded into several events.

The existing sentence only talked about input plugins, so it did not cover the <filter> case at all.

Note that the wording deliberately avoids saying "multiline does not work in filter". The regexp is compiled in multiline mode, so it can still match if the target field value already contains newline characters. What is impossible is joining separate events into one record — hence the pointer to fluent-plugin-concat.

Closes: fluent#338
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978
Watson1978 requested a review from kenhys July 24, 2026 06:37
Comment thread parser/multiline.md Outdated
The argument of `<filter>` is a tag pattern, so `<filter parser>` reads
as a filter for the tag "parser". Describe it as the `<parse>` section
under `<filter>` instead.

Also add a note to filter/parser.md. The existing wording says that you
can reuse the predefined formats there, which does not hold for
`multiline`.

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978
Watson1978 requested a review from kenhys July 24, 2026 07:45
@kenhys
kenhys merged commit 3dee451 into fluent:1.0 Jul 24, 2026
2 checks passed
@Watson1978
Watson1978 deleted the clarify-multiline-parser-scope branch July 24, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parser plugin multiline doesn't work as it appears in docs

2 participants