Docs feedback: It should be specified that each daemonset pod watches ALL pods logs in the cluster

[adrian-salas]

URL

https://grafana.com/docs/alloy/latest/configure/kubernetes/

Component(s)

No response

Feedback

For the context, we have had issues lately on our Azure cluster where it seemed that we reached AKS Inflight limits.

Azure Inflight determines if the Kube API is throttled.
Using Alloy on its default configuration on 4 cluster saturated the Kube API in a way that new objects couldn't even be created.

Since each alloy pod (we could have 20 nodes, so 20 pods) would watch all logs of the cluster, the path containerLogs was definitely saturated, and Azure Inflight too.

In the end, we could add the following rule for Alloy on our pod configuration:

        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          action        = "keep"
          regex         = env("K8S_NODE_NAME")
        }

with K8S_NODE_NAME being and extraEnv given to alloy

  extraEnv:
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

That configuration allows retrieval of pods logs that are on the same node as the alloy pods that watches

Firstly, there was a huge memory decrease, each pod was consuming ~3Go of memory per pod.
Now Alloy uses around 300Mo per pod.

The path for containerLogs had a night an day difference too:

And also, specific for Azure on our case, a decrease in Inlfight requests:

--
Consequences:

Since Loki deduplicates entries, we hadn't noticed that alloy was actually watching all pods each at first.

We have had 4 saturations on our Kubernetes API, that affected our pipelines (runners were throttled)
Also, our monitoring has been affected for a while, since we have had alloy disabled on multiple times.

Of course, there was time spend from our team as well as increased costs for our clusters since we tried changing AKS Inflight tier (around 300$ per cluster)

To summary, i think for the potential cost of that behavior, i think it should be better specified in the documentation that this happens.
Of course, using clustering mode for Alloy would mitigate the issue too.
Either way, all of what I described wasn't really clear for us when reading the documentation

Tip

_{React with 👍 if this issue is important to you.}

[kalleep]

Hey, whenever you run alloy as a deamonset you should always limit discovery data to the running node. The example is for a metrics pipeline but it's true for logs as well.

If you are running a daemonset and using loki.source.podlogs we also call this out . But we had a bug here that was recently fixed to make it work properly.

If you limit the discovery data you don't need that relabel rule. We call this out in several places:

• loki.source.podlogs
• discovery.kubernetes
• example for collecting logs in k8s

But there are some places we should update to use node filter:

• local.file_match example

What other places do you think this call out should be added?

[adrian-salas]

Hello @kalleep, thanks for the quick answer !

Now that I read back the links you sent:

• discovery.kubernetes
• example for collecting logs in k8s

and with the "issue" in my head, it seems pretty clear indeed.

But to be honest, I probably completely omitted the notices when installing Alloy the first time.

--
Knowing now what could happen if that configuration is missed (like I did), wouldn't it be better to at mention it here too ? (and probably have a link to discovery.kubernetes)

Also knowing what could be caused when missed, I guess that discovery.kubernetes could have more gravitas maybe ?

This is what i had in mind:

and

Also the wording "This configuration could be useful if you are running Alloy as a DaemonSet."
OMI, since Alloy is deployed as daemonset by default, is that it should be mandatory.

I know it's just details, but I'm sure I'm not the only one that missed that point when installing Alloy (especially since i posted on reddit to debug my problems, and the person who answered had the same issues, and just completely missed that too)

Just giving ideas that I think could help understand better the first configuration of Alloy

[kalleep]

No you are not the only person this happened to. @clayton-cornell any input on where we should update and make this clearer?

[clayton-cornell]

We can add admonitions to those topics... I would convert the first suggested note into a Caution... stands out more and suitable because there's a bigger impact if things are missed. The second suggested note... I think would be OK as a note... location... probably the same general area as suggested.

[clayton-cornell]

@kalleep Can this issue be closed as fixed in #4955 ?