fluentd_output_status_buffer_total_bytes metric returns negative (minus) results

[dannyk81]

We are running fluentd v1.2.2 and fluent-plugin-prometheus v1.0.1 and we've identified few servers that report negative numbers for the fluentd_output_status_buffer_total_bytes metric.

Element	Value
fluentd_output_status_buffer_total_bytes{cluster="abc",datacenter="xyz",env="prd",host="kubw-48",instance="x.x.x.x:24231",job="kubernetes-pods",k8s_app="fluentd-logging",kubernetes_namespace="kube-system",kubernetes_pod_name="fluentd-worker-35c9d",kubernetes_pod_node_name="kubw-48",node="kubw-48",plugin_id="object:3fe5d891e918",role="worker",type="forward"}	-19394669

I'm not sure if it's the problem of the plugin or fluentd itself, since I believe the plugin collects the metrics from fluentd monitoring agent.

[xidiandb]

I had the same problem

[rehevkor5]

It looks possible that this has been fixed in fluent/fluentd#2734 for fluent/fluentd#2712 released as v1.9.0 of fluentd.

[pranavmarla]

FYI, this is still not fixed as of Fluentd v1.11.2 -- see the linked issue above.

[Watson1978]

I am closing this issue as it is a duplicate of another report regarding negative buffer sizes.

I think this is not a bug in fluent-plugin-prometheus itself, but rather an issue with Fluentd core's internal buffer size calculations (often caused by multi-worker configuration issues or race conditions).

We have transferred the active discussion to the main Fluentd repository. Please follow fluent/fluentd#5303 for any further updates or to provide your configuration details.

Thank you!