parser: clarify that multiline parser does not join separate events#649
Merged
Conversation
Closes: fluent#338 Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
kenhys
requested changes
Jul 24, 2026
The argument of `<filter>` is a tag pattern, so `<filter parser>` reads as a filter for the tag "parser". Describe it as the `<parse>` section under `<filter>` instead. Also add a note to filter/parser.md. The existing wording says that you can reuse the predefined formats there, which does not hold for `multiline`. Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
kenhys
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #338
MultilineParser#parseinlib/fluent/plugin/parser_multiline.rbapplies the regexp built fromformatN(compiled withRegexp::MULTILINE) to the text it is given, then loops overRegexp::MatchData#post_matchand yields once per match. So a single call can emit several records, but every bit of state is local to that call: the plugin keeps no buffer across calls, and therefore cannot combine lines which arrive as separate events. It only exposeshas_firstline?/firstline?as helpers for the caller.The actual line buffering is in
lib/fluent/plugin/in_tail.rb: it detects the multiline mode with@multiline_mode = parser_config["@type"].include?("multiline"), andparse_multilinesappends lines to a buffer until@parser.firstline?(line)matches the next first line.lib/fluent/plugin/filter_parser.rbhas no guard againstmultiline: noConfigError, no warning. So a<filter>with@type parserand@type multilinein its<parse>section silently parses a single field of an already-split event, which is not what users expect. That is the confusion reported in #338. On top of that,filter_one_recordadds a new event peryield, so one input event can even be expanded into several events.The existing sentence only talked about input plugins, so it did not cover the
<filter>case at all.Note that the wording deliberately avoids saying "multiline does not work in filter". The regexp is compiled in multiline mode, so it can still match if the target field value already contains newline characters. What is impossible is joining separate events into one record — hence the pointer to
fluent-plugin-concat.