Skip to content

Latest commit

 

History

History
188 lines (136 loc) · 9.24 KB

README.md

File metadata and controls

188 lines (136 loc) · 9.24 KB

StatsD Receiver

Status
Stability beta: metrics
Distributions contrib
Issues Open issues Closed issues
Code coverage codecov
Code Owners @jmacd, @dmitryax

StatsD receiver for ingesting StatsD messages(https://github.com/statsd/statsd/blob/master/docs/metric_types.md) into the OpenTelemetry Collector.

Use case: it does not support horizontal pool of collectors. Desired work case is that customers use the receiver as an agent with a single input at the same time.

Configuration

The Following settings are optional:

  • endpoint: Address and port to listen on.

    • For udp and tcp based transport, this config will default to localhost:8125
    • For unixgram transport, this config will default to /var/run/statsd-receiver.sock

  • transport (default = udp): Protocol used by the StatsD server. Currently supported transports can be found in this file.

  • socket_permissions (default = 0622): When transport is set to unixgram, can be used to customize permissions of the binded socket.

  • aggregation_interval: 70s(default value is 60s): The aggregation time that the receiver aggregates the metrics (similar to the flush interval in StatsD server)

  • enable_metric_type: true(default value is false): Enable the statsd receiver to be able to emit the metric type(gauge, counter, timer(in the future), histogram(in the future)) as a label.

  • enable_ip_only_aggregation (default value is false): Enables metric aggregation on Client+IP only. Normally, aggregation is performed on Client+IP+Port. This setting is useful when the client sends metrics from a random ports or the receiver should aggregate metrics from the same client but different ports.

  • enable_simple_tags: true(default value is false): Enable parsing tags that do not have a value, e.g. #mykey instead of #mykey:myvalue. DogStatsD supports such tagging.

  • is_monotonic_counter (default value is false): Set all counter-type metrics the statsd receiver received as monotonic.

  • timer_histogram_mapping:(default value is below): Specify what OTLP type to convert received timing/histogram data to.

"statsd_type" specifies received Statsd data type. Possible values for this setting are "timing", "timer", "histogram" and "distribution".

"observer_type" specifies OTLP data type to convert to. We support "gauge", "summary", and "histogram". For "gauge", it does not perform any aggregation. For "summary, the statsD receiver will aggregate to one OTLP summary metric for one metric description (the same metric name with the same tags). By default, it will send percentile 0, 10, 50, 90, 95, 100 to the downstream. The "histogram" setting selects an auto-scaling exponential histogram configured with only a maximum size, as shown in the example below unless it matches the configured explicit_buckets matcher pattern. TODO: Add a new option to use a smoothed summary like Prometheus: #3261

Example:

receivers:
  statsd:
  statsd/2:
    endpoint: "localhost:8127"
    aggregation_interval: 70s
    enable_metric_type: true
    is_monotonic_counter: false
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "gauge"
      - statsd_type: "timing"
        observer_type: "histogram"
        histogram: 
          max_size: 100
          explicit_buckets:
            - matcher_pattern: "foo.*"
              buckets: [1, 10, 100]
            - matcher_pattern: "bar.*"
              buckets: [0.1, 0.5, 1]
      - statsd_type: "distribution"
        observer_type: "summary"
        summary: 
          percentiles: [0, 10, 50, 90, 95, 100]

The full list of settings exposed for this receiver are documented in config.go with detailed sample configurations in testdata/config.yaml.

Aggregation

Aggregation is done in statsD receiver. The default aggregation interval is 60s. The receiver only aggregates the metrics with the same metric name, metric type, label keys and label values. After each aggregation interval, the receiver will send all metrics (after aggregation) in this aggregation interval to the following workflow.

It supports: Counter(transferred to int):

  • statsdTestMetric1:3000|c|#mykey:myvalue statsdTestMetric1:4000|c|#mykey:myvalue (get the value after incrementation: 7000)
  • statsdTestMetric1:3000|c|#mykey:myvalue statsdTestMetric1:20|c|@0.25|#mykey:myvalue (get the value after incrementation with sample rate: 3000+20/0.25=3080)

When the receiver receives valid sample rate (greater than 0 and less than 1), we covert the count value to float, divide by the sample rate and then covert back to integer.

The official doc does not support negative counter, we follow this pattern at this time. There are some requests for negative counters, we need to take a look if we want to support later. For example: influxdata/telegraf#1898 https://thenewstack.io/collecting-metrics-using-statsd-a-standard-for-real-time-monitoring/ https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/#count

Gauge(transferred to double):

  • statsdTestMetric1:500|g|#mykey:myvalue statsdTestMetric1:400|g|#mykey:myvalue (get the latest value: 400)
  • statsdTestMetric1:500|g|#mykey:myvalue statsdTestMetric1:+2|g|#mykey:myvalue statsdTestMetric1:-1|g|#mykey:myvalue (get the value after calculation: 501)

Metrics

General format is:

<name>:<value>|<type>|@<sample-rate>|#<tag1-key>:<tag1-value>,<tag2-k/v>

Counter

<name>:<value>|c|@<sample-rate>|#<tag1-key>:<tag1-value>

It supports sample rate. When a sample rate is provided (between 0 and 1), the receiver divides the counter value by the sample rate during aggregation to estimate the actual count. For example, a counter value of 20 with sample rate 0.25 results in an estimated count of 80 (20 ÷ 0.25).

Note: Currently, the sample rate is applied during aggregation and the adjusted value is exported. If OTLP adds native support for sample rate as a metric parameter in the future, the implementation may be updated to preserve the original sample rate information.

Gauge

<name>:<value>|g|@<sample-rate>|#<tag1-key>:<tag1-value>

Timer

<name>:<value>|ms|@<sample-rate>|#<tag1-key>:<tag1-value> <name>:<value>|h|@<sample-rate>|#<tag1-key>:<tag1-value>

It supports sample rate.

Testing

Full sample collector config

receivers:
  statsd:
    endpoint: "localhost:8125" # default
    aggregation_interval: 60s  # default
    enable_metric_type: false   # default
    is_monotonic_counter: false # default
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "histogram"
        histogram:
          max_size: 50
          explicit_buckets:
            - matcher_pattern: "foo.*"
              buckets: [1, 10, 100]
            - matcher_pattern: "bar.*"
              buckets: [0.1, 0.5, 1]
      - statsd_type: "distribution"
        observer_type: "histogram"
        histogram: 
          max_size: 50    
      - statsd_type: "timing"
        observer_type: "summary"

exporters:
  file:
    path: ./test.json

service:
  pipelines:
    metrics:
     receivers: [statsd]
     exporters: [file]

Send StatsD message into the receiver

A simple way to send a metric to localhost:8125:

echo "test.metric:42|c|#myKey:myVal" | nc -w 1 -u -4 localhost 8125;
echo "test.metric:42|c|#myKey:myVal" | nc -w 1 -u -6 localhost 8125;

Which sends a UDP packet using both IPV4 and IPV6, which is needed because the receiver's UDP server only accepts one or the other.