Skip to content

Conversation

@wallrj
Copy link
Member

@wallrj wallrj commented Nov 20, 2024

I assume that the controller-runtime logs were being discarded.
With this change I hope to find some controller-runtime logs among the e2e test logs.

Testing

$ make test-e2e-gke
...
{
  "ts": 1732123466929.3813,
  "caller": "metrics/metrics.go:97",
  "msg": "cache not synced yet, skipping metrics venaficonnection_*",
  "source": "controller-runtime",
  "v": 0
}
...
{"ts":1732123472568.4834,"caller":"agent/run.go:409","msg":"Data sent successfully","v":0,"logger":"Run.gatherAndOutputData.postData"}

That single message seems to come from venafi-connection-lib, so perhaps it's misleading to label these logs with source: "controller-runtime":

@maelvls Does that log message indicate a bug in the way we're instantiating the venafi-connection-lib client?

Signed-off-by: Richard Wall <richard.wall@venafi.com>
// TODO(mael): The rest of the codebase uses the standard "log" package,
// venafi-connection-lib uses "go-logr/logr", and client-go uses "klog". We
// should standardize on one of them, probably "slog".
ctrlruntimelog.SetLogger(logr.Logger{})
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


extraArgs:
- --logging-format=json
- --log-level=2
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this so that could see trace level logs in the e2e test.
As it happens, the single controller-runtime log message from venafi-connection-lib is at level 0, so this change is not strictly necessary.

@maelvls
Copy link
Member

maelvls commented Nov 21, 2024

cache not synced yet, skipping metrics venaficonnection_*

I remember adding this log line in venafi-connection-lib's metrics.go (internal link). I remember writing this:

// We found a niche problem where `/metrics` would hang forever in case of a
// misconfigured RBAC. This was due to `cache.List` hanging until the cache
// is synced. To prevent that, we skip reporting this subset of the metrics.
if !hasSynced(cc.cache) {
	cc.log.Info("cache not synced yet, skipping metrics venaficonnection_*")
	return
}

I don't know why WaitForCache is taking so long to return. The Helm chart's RBAC is correct, so it shouldn't be this... This needs debugging, but we can do that in the next PR.

@wallrj wallrj merged commit eb3c30a into master Nov 21, 2024
@wallrj wallrj deleted the VC-36032/controller-runtime-logging branch November 21, 2024 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants