Skip to content

Handle YAML configuration format on configuration file - #3712

Merged
ashie merged 15 commits into
fluent:masterfrom
cosmo0920:revised-yaml-config
May 18, 2022
Merged

Handle YAML configuration format on configuration file#3712
ashie merged 15 commits into
fluent:masterfrom
cosmo0920:revised-yaml-config

Conversation

@cosmo0920

@cosmo0920 cosmo0920 commented Apr 11, 2022

Copy link
Copy Markdown
Contributor

Which issue(s) this PR fixes:
Fixes #554
Revised #2818

What this PR does / why we need it:

Recently, fluent-bit supports YAML configuration in fluent/fluent-bit#4621.

Now, it is reasonable to handle YAML configuration format on Fluentd, too.
This is because YAML config format is more kubernetes friendly and machine friendly.
It's time to discuss to support YAML configuration format again.

Docs Changes:
fluent/fluentd-docs-gitbook#402

Release Note:

Same as title

ganmacs and others added 6 commits April 8, 2022 14:57
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
@cosmo0920
cosmo0920 requested review from ashie and kenhys April 11, 2022 10:13
@cosmo0920 cosmo0920 self-assigned this Apr 11, 2022
Comment thread lib/fluent/config.rb Outdated
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
@cosmo0920

Copy link
Copy Markdown
Contributor Author

@ashie @kenhys How's it going to review this PR?

@ashie

ashie commented May 17, 2022

Copy link
Copy Markdown
Member

I think #3535 and this are featured functions of next release (v1.15).
#3535 has been landed just now, so I'll start to review this from now 🥲
(Although I already reviewed it roughly and it seems fine, I'll check the details more.)

Comment thread lib/fluent/command/fluentd.rb Outdated
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Comment thread test/test_supervisor.rb Outdated
Comment thread test/test_config.rb Outdated
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
@cosmo0920
cosmo0920 force-pushed the revised-yaml-config branch from 348c5e2 to 389bc72 Compare May 18, 2022 01:46
cosmo0920 and others added 2 commits May 18, 2022 10:48
Co-authored-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

Co-authored-by: Takuro Ashie <ashie@clear-code.com>
@cosmo0920
cosmo0920 force-pushed the revised-yaml-config branch from a68d096 to db8b245 Compare May 18, 2022 01:48
Comment thread lib/fluent/config/yaml_parser/parser.rb
@cosmo0920

Copy link
Copy Markdown
Contributor Author

Any concerns?

@ashie
ashie merged commit 12d445f into fluent:master May 18, 2022
@ashie

ashie commented May 18, 2022

Copy link
Copy Markdown
Member

👍

@cosmo0920

Copy link
Copy Markdown
Contributor Author

Thanks!!

@cosmo0920
cosmo0920 deleted the revised-yaml-config branch May 18, 2022 04:51
kavirajk referenced this pull request in grafana/loki Oct 10, 2023
…10839)

[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [fluentd](https://www.fluentd.org/)
([source](https://togithub.com/fluent/fluentd)) | `'1.14.2'` ->
`'1.15.3'` |
[![age](https://developer.mend.io/api/mc/badges/age/rubygems/fluentd/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/rubygems/fluentd/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/rubygems/fluentd/'1.14.2'/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/rubygems/fluentd/'1.14.2'/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

### GitHub Vulnerability Alerts

####
[CVE-2022-39379](https://togithub.com/fluent/fluentd/security/advisories/GHSA-fppq-mj76-fpj2)

### Impact
A remote code execution (RCE) vulnerability in non-default
configurations of Fluentd allows unauthenticated attackers to execute
arbitrary code via specially crafted JSON payloads.

Fluentd setups are only affected if the environment variable
`FLUENT_OJ_OPTION_MODE` is explicitly set to `object`.

Please note: The option FLUENT_OJ_OPTION_MODE was introduced in Fluentd
version 1.13.2. Earlier versions of Fluentd are not affected by this
vulnerability.

### Patches
v1.15.3

### Workarounds
Do not use `FLUENT_OJ_OPTION_MODE=object`.

### References

* GHSL-2022-067

---

### Release Notes

<details>
<summary>fluent/fluentd (fluentd)</summary>

###
[`v1.15.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1153---20221102)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.15.2...v1.15.3)

##### Bug Fix

-   Support glob for `!include` directive in YAML config format

[https://github.com/fluent/fluentd/pull/3917](https://togithub.com/fluent/fluentd/pull/3917)3917
-   Remove meaningless oj options

[https://github.com/fluent/fluentd/pull/3929](https://togithub.com/fluent/fluentd/pull/3929)3929
-   Fix log initializer to correctly create per-process files on Windows

[https://github.com/fluent/fluentd/pull/3939](https://togithub.com/fluent/fluentd/pull/3939)3939
-   out_file: Fix the multi-worker check with `<worker 0-N>` directive

[https://github.com/fluent/fluentd/pull/3942](https://togithub.com/fluent/fluentd/pull/3942)3942

##### Misc

-   Fix broken tests on Ruby 3.2

[https://github.com/fluent/fluentd/pull/3883](https://togithub.com/fluent/fluentd/pull/3883)3[https://github.com/fluent/fluentd/pull/3922](https://togithub.com/fluent/fluentd/pull/3922)ull/3922

###
[`v1.15.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1152---20220822)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.15.1...v1.15.2)

##### Enhancement

-   Add a new system configuration `enable_jit`

[https://github.com/fluent/fluentd/pull/3857](https://togithub.com/fluent/fluentd/pull/3857)3857

##### Bug Fix

-   out_file: Fix append mode with `--daemon` flag

[https://github.com/fluent/fluentd/pull/3864](https://togithub.com/fluent/fluentd/pull/3864)3864
-   child_process: Plug file descriptor leak

[https://github.com/fluent/fluentd/pull/3844](https://togithub.com/fluent/fluentd/pull/3844)3844

##### Misc

-   Drop win32-api gem to support Ruby 3.2

[https://github.com/fluent/fluentd/pull/3849](https://togithub.com/fluent/fluentd/pull/3849)3[https://github.com/fluent/fluentd/pull/3866](https://togithub.com/fluent/fluentd/pull/3866)ull/3866

###
[`v1.15.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1151---20220727)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.15.0...v1.15.1)

##### Bug Fix

-   Add support for concurrent append in out_file

[https://github.com/fluent/fluentd/pull/3808](https://togithub.com/fluent/fluentd/pull/3808)3808

##### Misc

-   in_tail: Show more information on skipping update_watcher

[https://github.com/fluent/fluentd/pull/3829](https://togithub.com/fluent/fluentd/pull/3829)3829

###
[`v1.15.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1150---20220629)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.6...v1.15.0)

##### Enhancement

-   in_tail: Add log throttling in files based on group rules

[https://github.com/fluent/fluentd/pull/3535](https://togithub.com/fluent/fluentd/pull/3535)3[https://github.com/fluent/fluentd/pull/3771](https://togithub.com/fluent/fluentd/pull/3771)ull/3771
-   Add `dump` command to fluent-ctl

[https://github.com/fluent/fluentd/pull/3680](https://togithub.com/fluent/fluentd/pull/3680)3680
-   Handle YAML configuration format on configuration file

[https://github.com/fluent/fluentd/pull/3712](https://togithub.com/fluent/fluentd/pull/3712)3712
- Add `restart_worker_interval` parameter in `<system>` directive to set
interval to restart workers that has stopped for some
reas[https://github.com/fluent/fluentd/pull/3768](https://togithub.com/fluent/fluentd/pull/3768)ull/3768

##### Bug fixes

-   out_forward: Fix to update timeout of cached sockets

[https://github.com/fluent/fluentd/pull/3711](https://togithub.com/fluent/fluentd/pull/3711)3711
- in_tail: Fix a possible crash on file rotation when `follow_inodes
true`

[https://github.com/fluent/fluentd/pull/3754](https://togithub.com/fluent/fluentd/pull/3754)3754
-   output: Fix a possible crash of flush thread

[https://github.com/fluent/fluentd/pull/3755](https://togithub.com/fluent/fluentd/pull/3755)3755
-   in_tail: Fix crash bugs on Ruby 3.1 on Windows

[https://github.com/fluent/fluentd/pull/3766](https://togithub.com/fluent/fluentd/pull/3766)3766
- in_tail: Fix a bug that in_tail cannot open non-ascii path on Windows

[https://github.com/fluent/fluentd/pull/3774](https://togithub.com/fluent/fluentd/pull/3774)3774
- Fix a bug that fluentd doesn't release its own log file even after
rotated by
external
to[https://github.com/fluent/fluentd/pull/3782](https://togithub.com/fluent/fluentd/pull/3782)ull/3782

##### Misc

-   in_tail: Simplify TargetInfo related code

[https://github.com/fluent/fluentd/pull/3489](https://togithub.com/fluent/fluentd/pull/3489)3489
-   Fix a wrong issue number in CHANGELOG

[https://github.com/fluent/fluentd/pull/3700](https://togithub.com/fluent/fluentd/pull/3700)3700
-   server helper: Add comments to linger_timeout behavior about Windows

[https://github.com/fluent/fluentd/pull/3701](https://togithub.com/fluent/fluentd/pull/3701)3701
-   service_discovery: Fix typo

[https://github.com/fluent/fluentd/pull/3724](https://togithub.com/fluent/fluentd/pull/3724)3724
-   test: Fix unstable tests and warnings

[https://github.com/fluent/fluentd/pull/3745](https://togithub.com/fluent/fluentd/pull/3745)3[https://github.com/fluent/fluentd/pull/3753](https://togithub.com/fluent/fluentd/pull/3753)u[https://github.com/fluent/fluentd/pull/3767](https://togithub.com/fluent/fluentd/pull/3767)t[https://github.com/fluent/fluentd/pull/3783](https://togithub.com/fluent/fluentd/pull/3783)l[https://github.com/fluent/fluentd/pull/3784](https://togithub.com/fluent/fluentd/pull/3784)n[https://github.com/fluent/fluentd/pull/3785](https://togithub.com/fluent/fluentd/pull/3785)f[https://github.com/fluent/fluentd/pull/3787](https://togithub.com/fluent/fluentd/pull/3787)com/fluent/fluentd/pull/3787

###
[`v1.14.6`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1146---20220331)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.5...v1.14.6)

##### Enhancement

-   Enable server plugins to specify socket-option `SO_LINGER`

[https://github.com/fluent/fluentd/pull/3644](https://togithub.com/fluent/fluentd/pull/3644)3644
-   Add `--umask` command line parameter

[https://github.com/fluent/fluentd/pull/3671](https://togithub.com/fluent/fluentd/pull/3671)3[https://github.com/fluent/fluentd/pull/3679](https://togithub.com/fluent/fluentd/pull/3679)ull/3679

##### Bug fixes

-   Fix metric name typo

[https://github.com/fluent/fluentd/pull/3630](https://togithub.com/fluent/fluentd/pull/3630)3[https://github.com/fluent/fluentd/pull/3673](https://togithub.com/fluent/fluentd/pull/3673)ull/3673
- Apply modifications in pipeline to the records being passed to
`@ERROR` label

[https://github.com/fluent/fluentd/pull/3631](https://togithub.com/fluent/fluentd/pull/3631)3631
-   Fix wrong calculation of retry interval

[https://github.com/fluent/fluentd/pull/3640](https://togithub.com/fluent/fluentd/pull/3640)3[https://github.com/fluent/fluentd/pull/3649](https://togithub.com/fluent/fluentd/pull/3649)u[https://github.com/fluent/fluentd/pull/3685](https://togithub.com/fluent/fluentd/pull/3685)t[https://github.com/fluent/fluentd/pull/3686](https://togithub.com/fluent/fluentd/pull/3686)luentd/pull/3686
-   Support IPv6 address for `rpc_endpoint` in `system` config

[https://github.com/fluent/fluentd/pull/3641](https://togithub.com/fluent/fluentd/pull/3641)3641

##### Misc

-   CI: Support Ruby 3.1 except Windows

[https://github.com/fluent/fluentd/pull/3619](https://togithub.com/fluent/fluentd/pull/3619)3619
-   Switch to GitHub Discussions

[https://github.com/fluent/fluentd/pull/3654](https://togithub.com/fluent/fluentd/pull/3654)3654
-   Fix CHANGELOG.md heading styles

[https://github.com/fluent/fluentd/pull/3648](https://togithub.com/fluent/fluentd/pull/3648)3648
-   Declare `null_value_pattern` as `regexp`

[https://github.com/fluent/fluentd/pull/3650](https://togithub.com/fluent/fluentd/pull/3650)3650

###
[`v1.14.5`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1145---20220209)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.4...v1.14.5)

##### Enhancement

-   Add support for "application/x-ndjson" to `in_http`

[https://github.com/fluent/fluentd/pull/3616](https://togithub.com/fluent/fluentd/pull/3616)3616
-   Add support for ucrt binary for Windows

[https://github.com/fluent/fluentd/pull/3613](https://togithub.com/fluent/fluentd/pull/3613)3613

##### Bug fixes

-   Don't retry when `retry_max_times == 0`

[https://github.com/fluent/fluentd/pull/3608](https://togithub.com/fluent/fluentd/pull/3608)3608
-   Fix hang-up issue during TLS handshake in `out_forward`

[https://github.com/fluent/fluentd/pull/3601](https://togithub.com/fluent/fluentd/pull/3601)3601
-   Bump up required ServerEngine to v2.2.5

[https://github.com/fluent/fluentd/pull/3599](https://togithub.com/fluent/fluentd/pull/3599)3599
-   Fix "invalid byte sequence is replaced" warning on Kubernetes

[https://github.com/fluent/fluentd/pull/3596](https://togithub.com/fluent/fluentd/pull/3596)3596
- Fix "ArgumentError: unknown keyword: :logger" on Windows with Ruby 3.1

[https://github.com/fluent/fluentd/pull/3592](https://togithub.com/fluent/fluentd/pull/3592)3592

###
[`v1.14.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1144---20220106)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.3...v1.14.4)

##### Enhancement

-   `in_tail`: Add option to skip long lines (`max_line_size`)

[https://github.com/fluent/fluentd/pull/3565](https://togithub.com/fluent/fluentd/pull/3565)3565

##### Bug fix

- Incorrect BufferChunkOverflowError when each event size is <
`chunk_limit_size`

[https://github.com/fluent/fluentd/pull/3560](https://togithub.com/fluent/fluentd/pull/3560)3560
- On macOS with Ruby 2.7/3.0, `out_file` fails to write events if
`append` is true.

[https://github.com/fluent/fluentd/pull/3579](https://togithub.com/fluent/fluentd/pull/3579)3579
-   test: Fix unstable test cases

[https://github.com/fluent/fluentd/pull/3574](https://togithub.com/fluent/fluentd/pull/3574)3[https://github.com/fluent/fluentd/pull/3577](https://togithub.com/fluent/fluentd/pull/3577)ull/3577

###
[`v1.14.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1143---20211126)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.2...v1.14.3)

##### Enhancement

-   Changed to accept `http_parser.rb` 0.8.0.
    `http_parser.rb` 0.8.0 is ready for Ractor.

[https://github.com/fluent/fluentd/pull/3544](https://togithub.com/fluent/fluentd/pull/3544)3544

##### Bug fix

-   in_tail: Fixed a bug that no new logs are read when
    `enable_stat_watcher true` and `enable_watch_timer false` is set.

[https://github.com/fluent/fluentd/pull/3541](https://togithub.com/fluent/fluentd/pull/3541)3541
-   in_tail: Fixed a bug that the beginning and initial lines are lost
after startup when `read_from_head false` and path includes wildcard
'\*'.[https://github.com/fluent/fluentd/pull/3542](https://togithub.com/fluent/fluentd/pull/3542)/3542
-   Fixed a bug that processing messages were lost when
    BufferChunkOverflowError was thrown even though only a specific
message size exceeds
chunk_limi[https://github.com/fluent/fluentd/pull/3553](https://togithub.com/fluent/fluentd/pull/3553)t[https://github.com/fluent/fluentd/pull/3562](https://togithub.com/fluent/fluentd/pull/3562)luentd/pull/3562

##### Misc

-   Bump up required version of `win32-service` gem.
newer version is required to implement additional `fluent-ctl` commands.

[https://github.com/fluent/fluentd/pull/3556](https://togithub.com/fluent/fluentd/pull/3556)3556

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no
schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/grafana/loki).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy44LjEiLCJ1cGRhdGVkSW5WZXIiOiIzNy44LjEiLCJ0YXJnZXRCcmFuY2giOiJtYWluIn0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@daipom

daipom commented Apr 15, 2024

Copy link
Copy Markdown
Contributor

#4463 (comment)

it appears that YAML format doesn't support the $log_level

If someone remembers something about this, please let me know.

if (v = config.delete('$id'))
sb.add_line('@id', v)
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daipom Probably, we should add handler for $log_level here. Do you have a bandwidth for this?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I don't think this is a serious problem.
I will work on other issues for a while.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Fair enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Configuration as yaml/json

4 participants