[processor/cumulativetodelta] staleness timeout should be enabled by default

[odubajDT]

Component(s)

processor/cumulativetodelta

Is your feature request related to a problem? Please describe.

When using the processor with the default values, the max_staleness parameter is by default set to 0, which according to documentation keeps the metrics data indefinitely, even if there are no new metric datapoints coming through the processor anymore (for example from a killed Pod in k8s). Such behavior on a big environments leads to a steady increase in CPU and memory consumption, where the collector eventually will reach the limit and die.

Describe the solution you'd like

Default value of max_staleness parameter is set to 1h, which will avoid increasing memory consumption when processing large amounts of different metrics. The change of default behavior should be handled via feature gate.

With the parameter enabled, the memory and CPU usage remain stable.

Describe alternatives you've considered

Currently, there is only the alternative to set the timeout manually

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• processor/cumulativetodelta: @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[evan-bradley]

@TylerHelmuth for some context, we encountered this issue when investigating a customer setup recently, and determined that retaining all points forever by default feels like a memory leak. 5 minutes feels like a good default without any data to back that up, but I'd also be okay going even as long as an hour given the existing behavior doesn't seem to be regularly causing issues.

[odubajDT]

potentially we can speak about increasing the interval to 25h to cover the "once per day" export, as these kind of use-cases might be affected.

[evan-bradley]

Agreed, after looking into it more, I think 25 hours (24 hours plus 1 hour for export delays) is probably a safer default.

[mx-psi]

We have similar functionality in the Datadog exporter (that we are looking to deprecate in favor of the cumulative to delta processor). We have a default of 1 hour

opentelemetry-collector-contrib/pkg/datadog/config/config.go

Line 368 in e053df5

DeltaTTL: 3600,

and we have not heard complaints from any users about this default being too low

[braydonk]

The first time I ever ran into this was with a Prometheus receiver issue where the problem ended up being cumulativetodelta: #31591

(I did a talk where I went over the research to determine the processor was at fault: https://youtu.be/qMxxjB4meXo?t=261)