Cloudwatch metrics are too low

[juissi-t]

What's wrong?

We've been migrating from YACE to using Alloy with the embedded cloudwatch exporter. We are seeing an issue with the exported metrics: the values for the latest datapoint are not the same as the metrics in Cloudwatch (bug in YACE). In our YACE config, we had a delay for each metric for a few minutes, which would allow the metric datapoint value to stabilize in Cloudwatch, and we would then store the stable value to Mimir.

As such, I do not think setting the delay to 0 in the exporter config is the correct thing to do - the latest datapoint (and often the second-to-last one) are missing values, which makes the averages and sums be too low, regardless of the length and the delay parameters used. Having a delay is the only way we have so far found that allow us to fetch accurate metrics, so I believe this is a bug in Alloy.

Steps to reproduce

1. Fetch some volatile metric from Cloudwatch using Alloy's Cloudwatch exporter. I've used Lambda invocations for a function that is invoked ~3k times per second, with period of 1 minute, length of 5 minutes and scrape interval of 1 minute.
2. Observe both the actual Cloudwatch metric and the one imported using the exporter, e.g. in a same dashboard. Notice that the imported metric is never quite the same than the one from Cloudwatch. The imported metric in the screenshot is offset by -2 minutes to more closely align with the shape of the Cloudwatch metric.

3. The latest value from Cloudwatch is low, but it grows for more than a minute, until it reaches a stable value that no longer changes.

4. This could be solved with changing the delay parameter of the metric to 2 minutes and using Cloudwatch timestamps. For some other metrics, 5 minute delay could be more suitable.

System information

Linux amd64 and arm64

Software version

Grafana Alloy 1.7.1

Configuration

prometheus.exporter.cloudwatch "default" {
	sts_region = "eu-central-1"

	discovery {
		type    = "AWS/Lambda"
		regions = ["eu-central-1"]

		metric {
			name       = "Invocations"
			statistics = ["Sum"]
			period     = "1m"
			length     = "5m"
		}
	}
}

prometheus.scrape "cloudwatch_exporter" {
	clustering {
		enabled = true
	}

	targets         = prometheus.exporter.cloudwatch.default.targets
	scrape_interval = "1m"
	forward_to      = [prometheus.relabel.cloudwatch_exporter_relabel.receiver]
}

Logs

[juissi-t]

I graphed two different Cloudwatch metrics using 1 minute step and 2 minute offset for the imported metrics, and this issue seems to be apparent with at least the sum statistic for a volatile metric. SQS queue size average metrics match exactly, but the sum metric value for lambda invocations is consistently 100-200 units lower than the actual metric in Cloudwatch.

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[tmeijn]

Hey @juissi-t did you ever managed to solve this issue with Grafana Alloy? We are currently using Grafana Alloy to scrape CloudWatch metrics, but are facing the same issue. We actually are considering migrating to the plain YACE exporter, but I'd rather not do that...

[juissi-t]

No, this is still an issue for us. We migrated away from YACE to consolidate our tooling, but may need to migrate back.

[tmeijn]

Thanks for your reply @juissi-t. Have you and your team done anything else to partially mitigate this issue, like for example setting the period to 5m minutes? Then at least 3-4 minutes out of 5 minute period should've stabilized.

Hey @cristiangreco, @thepalbi hope you do not mind me mentioning y'all directly: I found y'all on the Git blame and found the original pull request here: grafana/agent#2740

Do you remember why y'all decided not to expose the delay parameter to the Alloy configuration? Currently the Alloy version of YACE is massively under reporting metrics and thus basically cannot be fully trusted. The recommended way in YACE is to add a delay, but not having this option exposed makes it impossible to implement this best practice. We would be happy to contribute, but I cannot clearly determine why this was not exposed in the first place and would like to know if there is some Chesterston Fence we are not aware of.

[cristiangreco]

I can't remember off the top of my head, but if you're open to submit a PR somebody from the team could review it.

[tmeijn]

Hey @juissi-t wondering if your team managed to test out this change and if it worked for you?