output: add support for new storage.overflow_action property

[edsiper]

When input plugins uses a filesystem based storage and the output plugin sets a storage.total_limit_size, as of now the default action when the buffer fills up is to drop the oldest chunk from the list, either from the backlog storage of from the plugin queue it self.

The following pull request, extends the functionality by allowing output pliugins to register a new behavior through the configuration option storage.overflow_action. This new configuration can take two values:

• drop_oldest_chunk: remove the oldest chunk to make room for new data. This is the default behavior.

• pause_ingestion: once the buffer 'almost fills up', pause the ingestion for plugins that are sending data on that route.

The following is a configuration example that pause ingestion:

service:
  flush: 1
  log_level: info
  http_server: true
  storage.path: ./storage

pipeline:
  inputs:
    - name: tail
      path: ~/logs/20MB.log
      read_from_head: true
      storage.type: filesystem

  outputs:
    - name: forward
      match: '*'
      host: 127.0.0.1
      port: 24224
      retry_limit: false
      storage.total_limit_size: 10M
      storage.overflow_action: pause_ingestion

When pause_ingestion is used, the ingestion will be paused when the queue is on >= 90% of it capacity, or if it has less than 5MB of space available.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[lecaros]

I'm seeing something, I'm not sure it's expected.
To repro: run your test twice without deleting the storage folder. Ensure the tail is paused. [ info] [input] pausing tail.0
Take note of the files in the storage folder (or just look for the pid part of it).
Then run it a third time. In the logs you'll see that all the files from storage were registered ( [ info] [input:storage_backlog:storage_backlog.1] register tail.0/66960-1724365479.232668000.flb); however, some of them are deleted.
Is this expected behavior @edsiper?