BatchProcessor limit data point rather than metrics

[kisieland]

Is your feature request related to a problem? Please describe.
Google Cloud Exporter communicate with an API that has a hard limit on number of data points sent in one request.
Introducing a Batch Processor before it can lead to situations in which the number of sent metrics is smaller than the limit but, the number of data points is above the limit, effectively dropping some of the data.

Describe the solution you'd like
I am proposing a new key (count_metric_data_points) in the BatchProcessor, that would be a boolean, which defaults to false.
False would correspond to current behaviour, whereas true would make processor count and limit data points rather than metrics.

Describe alternatives you've considered
I've considered introducing the limiting and batching in the exporter itself, but this would introduce a lot of custom logic that is already present in batch processor.

[kisieland]

@dashpole PTAL

[dashpole]

https://github.com/open-telemetry/opentelemetry-collector/issues/1139 seems similar, and proposes limiting by the number of bytes. The only thing i'm worried about is that the config field only applies to metrics, but the processor can be used by traces and logs (in the future) as well.

[kisieland]

WDYT about count_data_points field that would also include traces and count spans?

[dashpole]

That makes sense, except we are already counting spans for the existing limit. So it makes some sense, but is the same as the existing behavior.

[kisieland]

So, I guess we should stuck to original idea with: count_metric_data_points field

[dashpole]

@dyladan @bogdandrutu The only other option I can think of would be to change the current limit to use the number of points instead of the number of metrics. Was there a particular reason you chose to limit the number of metrics instead of the number of metric points? At first glance, it seems like limiting the number of points would be a more effective way to limit the metric payload than limiting the number of metrics...

[dashpole]

Notes from the meeting:

• We can go ahead and change from number of metrics to number of data points, as data points makes more sense.
• Since the name isn't specific to metrics vs data points, we can change the behavior of the field to limit the number of data points.
• This is mostly backwards-compatible, since the number of data points is at least the number of metrics. So this change could result in sending less data, but not more.
• When we make the change, we need to add a release note.

In the future, this should be an exporter helper instead of a processor. This will be easier to use. Some examples are that processors after the batch processor could add data points, which could result in errors. Also, when using multiple exporters in a pipeline, there may not be a single value that is ideal for all exporters.