Add Memory Limiting Capability to Service

[pjanotti]

Memory Limiting

Due to the high variability inherent to telemetry data it is hard to estimate the memory consumption when running the agent or collector. In practice users have to allocate the maximum amount of memory that will be available for each instance. On the steady state the memory consumption is expected to be low but in any case the service needs to gracefully handle when data is not flowing as intended. In this case queues/buffers are expected to fill but should be kept under certain limits to avoid OOM crashes (that cause all data on memory to be lost).

This requires:

1. Tracking memory usage and having configurable limits;
2. Capability to suspend data ingress when the limits are about to be reached;

Tracking Memory Usage

There should a periodic check of current memory usage against the desired limits. When the limits are about to be reached/crossed the tracking system needs to be able to notify receivers that they should suspend data ingestion.

Capability to Suspend Ingest

Receivers will in a cooperative fashion check if it is ok to add more data to their pipelines. This is cooperative so they can provide the proper response according to their protocol and if they should exert back pressure or not (this should be a configurable option on each receive).

Implementation Plan

• Modify receivers interface to support cooperative check if it is ok to
  ingest more data.
• Modify each receiver implementation on core to properly react when it the
  ingestion check informs that it should not ingest more data. Add proper
  configurations regarding back-pressure.
• Add memory limiter that can be used to notify receivers that they should
  suspend ingestion.

[pjanotti]

/cc @songy23 @tigrannajaryan @owais

[tigrannajaryan]

@pjanotti this can possibly be combined with general throttling capability. Throttling can be triggered by several conditions:

• When memory usage approaches the limit (what you described).
• When destination servers signal backpressure to exporters.
• Probably some other cases (e.g. when CPU usage is too high).

Throttling may be applied at different points in the system

• Receivers (what you described).
• Queues in the pipeline (can start dropping).
• Sending queues (what we planned to have in the future).

It requires further thinking to come up with coherent design of comprehensive throttling capability. This issue is a good start.

[bogdandrutu]

As @tigrannajaryan suggested I think this needs a proper design document before we jump into implementation. I would suggest to not push more changes until we agree on the overall design.

[flands]

memory_limiter processor exists today so closing this.