Skip to content

core: filesystem access abstraction layer addition#7564

Open
leonardo-albertovich wants to merge 317 commits intomasterfrom
leonardo-master-filesystem-access-abstraction-later
Open

core: filesystem access abstraction layer addition#7564
leonardo-albertovich wants to merge 317 commits intomasterfrom
leonardo-master-filesystem-access-abstraction-later

Conversation

@leonardo-albertovich
Copy link
Contributor

This PR adds a minimal filesystem abstraction layer mostly meant to allow the tail input plugin to ingest more than 8192 files while at the same time taking the platform specific code out of it so it can be leveraged by other components in order to remove duplicated code.

@leonardo-albertovich leonardo-albertovich marked this pull request as draft July 3, 2023 17:48
Leonardo Alminana and others added 16 commits July 17, 2023 10:20
Signed-off-by: Leonardo Alminana <leonardo@calyptia.com>
Co-authored-by: igorpeshansky <igorpeshansky@users.noreply.github.com>
Signed-off-by: leonardo-albertovich <leonardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
* workflows: enable sarif output for CIFuzz

This will make it easier to interpret the output of CIFuzz by making it
possible to view the result in the Github security page.

Signed-off-by: David Korczynski <david@adalogics.com>

* workflows: cifuzz: add sarif category

Signed-off-by: David Korczynski <david@adalogics.com>

---------

Signed-off-by: David Korczynski <david@adalogics.com>
Signed-off-by: David Korczynski <david@adalogics.com>
Signed-off-by: David Korczynski <david@adalogics.com>
`tmpfile()` can return NULL and this is not checked for at the moment.
If indeed it returns NULL then the call on line 70 `fileno(stdoutp)`
will cause a NULL dereference.

Signed-off-by: David Korczynski <david@adalogics.com>
Signed-off-by: David Korczynski <david@adalogics.com>
In the previous version a new option called net.max_worker_connections was introduced
to provide control around the maximum number of allowed TCP connections per worker inside
an output plugin, the logic was wrong.

In the upstream logic, connections are linked to one of these lists:

  - av_queue  : connections ready to be used (available)
  - busy_queue: connections that are busy (someone is using them)
  - drop_queue: connections in the cleanup phase (to be drop)

Fluent Bit don't create connections ahead of time, just on demand. When
a connection is created is placed into the busy_queue, when is not longer
needed one of these things happen:

  - if keepalive is enabled (default), the connection is moved to the 'av_queue'.
  - if keepalive is disabled, the connection is moved to 'drop_queue' then is
    closed and destroyed.

Based on the logic described above, to limit the number of total connections
in the worker, we only need to count the number of connections linked into
the 'busy_queue' list because if there are connections available 'av_queue' it
won't create a one.

This patch fixes the logic by only using the busy_queue to count the limit.

Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](helm/kind-action@v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
For the first revision, we'll provide log_key and one-by-one line JSON version
like as out_loki plugin.

---------

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Implement gzip compression

Signed-off-by: Catherine Fang <yinghongfang@google.com>
Co-authored-by: igorpeshansky <igorpeshansky@users.noreply.github.com>
Signed-off-by: braydonk <braydonk@google.com>
Signed-off-by: Phillip Whelan <phil@calyptia.com>
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: edsiper <edsiper@users.noreply.github.com>
Signed-off-by: Josh Baird <jbaird@galileo.io>
Leonardo Alminana added 2 commits October 5, 2023 09:18
Signed-off-by: Leonardo Alminana <leonardo@calyptia.com>
Signed-off-by: Leonardo Alminana <leonardo@calyptia.com>
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:18 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:18 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:18 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:44 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich marked this pull request as ready for review October 5, 2023 09:01
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 09:01 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 09:08 — with GitHub Actions Inactive
Signed-off-by: Leonardo Alminana <leonardo@calyptia.com>
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 11:35 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 12:00 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Jan 4, 2024

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Jan 4, 2024
@github-actions github-actions bot removed the Stale label Aug 16, 2024
@edsiper edsiper added this to the Fluent Bit v3.2.0 milestone Aug 19, 2024
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.