Improvement: Documentation for multi process worker could be improved to explain the behavior for hybrid config (defined inside worker directive and outside as well)

[patidar-suresh]

I was trying out multi process worker for one of my use case where few pipeline need to run in dedicated worker while some pipeline need to be defined outside any worker directive and should run on any of the worker process. While doing this, I wasn't sure how the configuration were included/loaded in each worker process and was unclear on the behavior observed. After doing some POC, I could understand the behavior. I suggest adding this in the documentation sot that it might help others understand the working behavior quickly.

The POC was done using "http" plugin which supports multi process worker and created pipelines to use it in dedicated workers as well as running in all the workers. The messages were sent using curl command on different ports to see the output.

Observations from POC:

Message forwarding behavior:
o For same port exposed in different workers, messages are sent randomly to one of the workers exposing that port.
o For same port exposed in worker(s) and at global (outside any worker), messages are sent randomly to one of the workers from entire worker list (irrespective of the worker exposing that port or not)
o For unique ports exposed in different workers, messages are sent to specific worker exposing respective ports

Filtering/Matching behavior:
o Common config (input/filters/match) which is not defined in any worker is included/considered in each worker.
o Common configs(input/filters/match) are included prior to any other config defined under specific worker tag

Following diagram depicts the actual setup used for POC.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[Watson1978]

Thanks for the detailed report, including the POC and the diagram. Sorry that this went unanswered for so long.

Your observations match the implementation. I checked them against v1.19.3.

Why configuration outside <worker> runs in every worker

In RootAgent#configure, elements inside a <worker N> directive get a target worker id assigned, and elements written outside any <worker> directive keep an empty target worker id list. The check used when walking the configuration is:

def for_another_worker?
  !@target_worker_ids.empty? && !@target_worker_ids.include?(Fluent::Engine.worker_id)
end

Since the list is empty for sections outside <worker>, for_another_worker? is always false for them, so they are never skipped and are configured in every worker.

Why common configuration is processed first

The <worker> handling appends the children of each <worker> directive to the root configuration and then removes the <worker> wrapper:

conf += e
...
conf.elements.delete_if{|e| e.name == 'worker'}

Because the children are appended, sections inside a <worker> directive always end up after the sections written outside it, regardless of where the <worker> directive appears in the file.

This has a consequence which is worth stating explicitly, and which your report hints at but does not spell out. <match> and <filter> are evaluated in order of definition and the first match wins. So a broad pattern written outside <worker> takes precedence over a pattern written inside <worker>, even when the <worker> directive comes earlier in the file. In other words, the textual order in the configuration file is not the effective order once <worker> is involved.

About the port observations

The behaviour you saw for ports follows from the first point. When a source is defined outside <worker>, it is instantiated in every worker, so every worker is listening on that port. That is why messages are distributed across the whole worker list rather than only the worker which also declares that port. The actual distribution across the listening sockets is handled by the kernel through SO_REUSEPORT, which is outside the scope of the configuration loading described above.

What we will document

The current article shows a hybrid configuration in the <worker> directive example, and the inline comments imply that the section outside <worker> runs on all workers. However, this is never stated as a rule, and the ordering behaviour is not mentioned at all. We plan to add a short subsection covering the following:

• Sections outside a <worker> directive are configured in every worker.
• Sections inside a <worker> directive are moved after the sections outside it, so the effective evaluation order differs from the order in the file.
• As a result, a broad <match> or <filter> written outside <worker> takes precedence over one written inside <worker>.

Thanks again for taking the time to investigate this and write it up.