engine: use effective post-processor counts for output metrics (v4.2)

[edsiper]

This PR fixes output counter parity when processors modify or drop logs.

It introduces route-effective record/byte tracking per task route, stores those effective values at output flush return, and makes engine output/router metrics use those effective counts instead of original task totals. As a result, proc_records, dropped_records, and retry-related counters reflect what was actually processed after output processors.

It also adds a runtime test that validates an output Lua processor dropping records leads to stdout.0 metrics showing proc_records_total=0 and dropped_records_total > 0.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch 4.2-log-counting

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37aeee63ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Remove unreachable dropped-record expectation

This check assumes fluentbit_output_dropped_records_total increases when the Lua output processor drops all logs, but that path short-circuits with FLB_OK for zero-record chunks in output_pre_cb_flush (include/fluent-bit/flb_output.h) and handle_output_event only increments dropped counters on FLB_RETRY/FLB_ERROR (src/flb_engine.c). With this test's stdout + drop-all Lua setup, dropped_records_total remains 0, so the condition never becomes true and the test can only timeout/fail.

Useful? React with 👍 / 👎.