Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: filesystem access abstraction layer addition #7564

Open
wants to merge 316 commits into
base: master
Choose a base branch
from

Conversation

leonardo-albertovich
Copy link
Collaborator

This PR adds a minimal filesystem abstraction layer mostly meant to allow the tail input plugin to ingest more than 8192 files while at the same time taking the platform specific code out of it so it can be leveraged by other components in order to remove duplicated code.

@leonardo-albertovich leonardo-albertovich marked this pull request as draft July 3, 2023 17:48
edsiper and others added 16 commits July 17, 2023 10:20
Co-authored-by: igorpeshansky <[email protected]>
Signed-off-by: leonardo-albertovich <[email protected]>
* workflows: enable sarif output for CIFuzz

This will make it easier to interpret the output of CIFuzz by making it
possible to view the result in the Github security page.

Signed-off-by: David Korczynski <[email protected]>

* workflows: cifuzz: add sarif category

Signed-off-by: David Korczynski <[email protected]>

---------

Signed-off-by: David Korczynski <[email protected]>
`tmpfile()` can return NULL and this is not checked for at the moment.
If indeed it returns NULL then the call on line 70 `fileno(stdoutp)`
will cause a NULL dereference.

Signed-off-by: David Korczynski <[email protected]>
Signed-off-by: David Korczynski <[email protected]>
In the previous version a new option called net.max_worker_connections was introduced
to provide control around the maximum number of allowed TCP connections per worker inside
an output plugin, the logic was wrong.

In the upstream logic, connections are linked to one of these lists:

  - av_queue  : connections ready to be used (available)
  - busy_queue: connections that are busy (someone is using them)
  - drop_queue: connections in the cleanup phase (to be drop)

Fluent Bit don't create connections ahead of time, just on demand. When
a connection is created is placed into the busy_queue, when is not longer
needed one of these things happen:

  - if keepalive is enabled (default), the connection is moved to the 'av_queue'.
  - if keepalive is disabled, the connection is moved to 'drop_queue' then is
    closed and destroyed.

Based on the logic described above, to limit the number of total connections
in the worker, we only need to count the number of connections linked into
the 'busy_queue' list because if there are connections available 'av_queue' it
won't create a one.

This patch fixes the logic by only using the busy_queue to count the limit.

Signed-off-by: Eduardo Silva <[email protected]>
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](helm/kind-action@v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
For the first revision, we'll provide log_key and one-by-one line JSON version
like as out_loki plugin.

---------

Signed-off-by: Hiroshi Hatake <[email protected]>
Implement gzip compression

Signed-off-by: Catherine Fang <[email protected]>
Co-authored-by: igorpeshansky <[email protected]>
Signed-off-by: GitHub <[email protected]>
Co-authored-by: edsiper <[email protected]>
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:18 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:18 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:18 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 07:44 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich marked this pull request as ready for review October 5, 2023 09:01
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 09:01 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 09:08 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 11:28 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to integration October 5, 2023 11:35 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr October 5, 2023 12:00 — with GitHub Actions Inactive
Copy link
Contributor

github-actions bot commented Jan 4, 2024

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Jan 4, 2024
@github-actions github-actions bot removed the Stale label Aug 16, 2024
@edsiper edsiper added this to the Fluent Bit v3.2.0 milestone Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.