Reduce misleading log noise from benign/unavailable integrations#327
Open
gecube wants to merge 1 commit into
Open
Reduce misleading log noise from benign/unavailable integrations#327gecube wants to merge 1 commit into
gecube wants to merge 1 commit into
Conversation
On hosts where some integrations are simply not present (e.g. Bottlerocket: no Docker, memory-only journald, SELinux-protected systemd bus, VPC CNI instead of Cilium) the node-agent logs a lot of errors and warnings that do not indicate any real failure. This sends operators down the wrong path when debugging unrelated problems, and the repeated systemd/SELinux warnings in particular consume the global log rate-limiter budget, hiding genuinely useful messages. Adjust these to reflect that they are expected, benign conditions: - cilium: log the missing eBPF map name and that Cilium is assumed unused, instead of a bare "no such file or directory". - container runtime / journald probes: an unavailable integration is normal (only one runtime is present on a node), so log at info level. - systemd: reading unit properties can be permanently denied by the host SELinux policy. Cache the failure per unit so the private bus is not re-queried and the identical warning is logged only once instead of on every scan. - journald: a missing or empty journal path is an expected probe outcome, log at info level. - aws metadata: a missing optional field (e.g. public-ipv4 on a private instance) returns a non-200 response; that is expected, not an error. No behavior changes other than logging; metric collection is unaffected. Signed-off-by: George Gaál <gb12335@gmail.com>
gecube
force-pushed
the
log-hygiene-benign-integrations
branch
from
b43c12b to
eb1ac05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the log-noise part of #914.
On hosts where some integrations simply aren't present (Bottlerocket: no Docker, memory-only journald, SELinux-protected systemd bus; AWS VPC CNI instead of Cilium; private instances without a public IP), the node-agent emits a stream of errors/warnings that don't indicate any real failure. This misleads operators debugging unrelated issues, and the repeated systemd/SELinux warnings in particular burn through the global log rate-limiter budget (
--log-per-second, default 10), silently dropping genuinely useful lines.This PR only adjusts logging — no flags, no behavior change, metric collection is unaffected:
no such file or directory.public-ipv4on a private instance) returns a non-200 response; that is expected, not an error.A follow-up PR will add opt-in flags (
--no-docker/--no-systemd/--no-journald) to fully disable irrelevant integrations.