Add prometheus pipeline translator under opentelemetry config by mitali-salvi · Pull Request #2148 · aws/amazon-cloudwatch-agent

mitali-salvi · 2026-06-09T16:05:08Z

Description

Adds support for opentelemetry.collect.prometheus configuration that reads an external Prometheus scrape config file and generates an OTel prometheus receiver pipeline feeding into the shared base metrics pipeline.

Config

{
  "opentelemetry": {
    "collect": {
      "prometheus": {
        "config_path": "/opt/aws/prometheus.yml",
        "cluster_name": "my-cluster"
      }
    }
  }
}

Field	Required	Description
config_path	Yes	Path to external Prometheus scrape config YAML
cluster_name	No	EKS cluster name (sets `k8s.cluster.name` resource attribute)

Behavior

References external YAML (open-ended scrape config, cannot be validated by JSON schema)
Translator auto-detects OTel wrapper format vs plain Prometheus format
When cluster_name is set, adds a transform processor to tag metrics with k8s.cluster.name
Pipeline name: metrics/otel_prometheus (avoids conflict with V1 metrics/prometheus/*)
Pipeline: prometheus receiver → forward/opentelemetry connector → shared base metrics pipeline (resourcedetection, batch, otlphttp)

Known limitation

Prometheus relabel configs using $1, $2 capture group references can conflict with the OTel confmap expandconverter which interprets $ as environment variable substitution. Documented in code comment.

Tests

Unit tests

TestPrometheusTranslator — nil conf, missing key, valid config, cluster_name, invalid cluster_name, missing file
TestPrometheusReceiverTranslator — OTel format parsing
TestPrometheusReceiverTranslatorPlainFormat — plain Prometheus format (global + scrape_configs)
TestPrometheusReceiverTranslatorMissingFile — file not found error
TestPrometheusTranslatorClusterNameProcessor / NoClusterNameProcessor — conditional processor
TestPrometheusOtelPipelineConfig — golden file test (end-to-end translation)

EC2 integration test (us-west-2, `mitsalvi-otel-test`)

Deployed custom agent binary to t3.medium instance
Prometheus receiver scraping node_exporter on localhost:9100
Metrics successfully exported to CloudWatch Monitoring endpoint ✅

EKS integration test (us-east-1, `cwa-otel-test`)

Deployed custom agent as pod with prometheus scrape config targeting kubernetes API server
Config with cluster_name: "cwa-otel-test"
k8s.cluster.name resource attribute applied via transform processor ✅
Metrics successfully exported to CloudWatch Monitoring endpoint ✅

Linting

make fmt ✓ | make lint ✓ (only pre-existing plugins/plugins.go impi issue) | go build ✓

Adds support for opentelemetry.collect.prometheus with a mandatory config_path field pointing to an external Prometheus scrape config YAML. The translator auto-detects OTel wrapper format vs plain Prometheus format and supports target_allocator with TLS. The pipeline feeds into the shared base metrics pipeline via forward connector. Config: {"opentelemetry": {"collect": {"prometheus": {"config_path": "/path/to/prometheus.yml"}}}}

…and tests - Validate cluster_name to prevent OTTL injection (alphanumeric, hyphens, dots, underscores only) - Add cluster_name to prometheus schema (was missing from pushed commit) - Add test cases: cluster_name processor, invalid cluster_name, missing config file, no-cluster-name path

JayPolanco · 2026-06-10T15:25:01Z

+service:
+    extensions: []
+    pipelines:
+        metrics/prometheus:


Any potential conflicts with V1 naming on this?

renamed to metrics/otel_prometheus to avoid any collision

- Rename pipeline from metrics/prometheus to metrics/otel_prometheus to avoid conflict with V1 legacy prometheus pipeline naming - Add test for plain prometheus format (global + scrape_configs) - Only inject default TLS paths for target_allocator if customer hasn't provided their own certs

The opentelemetry.collect.prometheus path should not assume operator deployment model. If target_allocator needs TLS, users should specify cert paths in their prometheus config explicitly.

Remove OTel wrapper format detection and target_allocator TLS injection. Only standard Prometheus format (scrape_configs at root) is supported. No format ambiguity, no error string matching.

okankoAMZ · 2026-06-09T20:51:24Z

+	// references (os.Expand), which can cause failures or empty replacements.
+	// This is a known limitation when prometheus configs with relabel_configs
+	// are loaded through the OTel config resolver pipeline.
+	if err := componentParser.Unmarshal(&cfg); err != nil {


so if it is a invalid prometheus config such as with a typo we assume it is an Otel config?

I removed OTel wrapper format support entirely in a previous commit. The translator now only accepts plain Prometheus format. If the config has a typo or is invalid YAML, it will fail rather than silently falling through to a different parsing path.

okankoAMZ · 2026-06-10T19:21:21Z

+	if clusterName, ok := common.GetString(conf, clusterNameKey); ok && clusterName != "" {
+		// Validate cluster_name to prevent OTTL injection (must be alphanumeric, hyphens, dots, underscores)
+		for _, c := range clusterName {
+			if (c < 'a' || c > 'z') && (c < 'A' || c > 'Z') && (c < '0' || c > '9') && c != '-' && c != '.' && c != '_' {


Can we use a regex check instead?

…name - Add filepath.Clean on config_path before use - Replace character-by-character loop with regexp for cluster_name validation - Add comment explaining fwdConnector in both Exporters and Connectors

okankoAMZ · 2026-06-10T19:55:38Z

+}
+
+func (t *prometheusReceiverTranslator) Translate(conf *confmap.Conf) (component.Config, error) {
+	factory := prometheusreceiver.NewFactory()


Why are we reinitializing the factory if we already created it in ID or vise versa?

mitali-salvi requested a review from a team as a code owner June 9, 2026 16:05

mitali-salvi added the ready for testing Indicates this PR is ready for integration tests to run label Jun 9, 2026

mitali-salvi added 2 commits June 9, 2026 20:18

Add comment about prometheus relabel dollar-sign escaping limitation

1bf4278

JayPolanco reviewed Jun 10, 2026

View reviewed changes

Comment thread translator/translate/otel/pipeline/opentelemetry/prometheus/translator_test.go

JayPolanco reviewed Jun 10, 2026

View reviewed changes

Comment thread translator/translate/otel/pipeline/opentelemetry/prometheus/translator.go Outdated

JayPolanco reviewed Jun 10, 2026

View reviewed changes

Comment thread translator/translate/otel/pipeline/opentelemetry/prometheus/translator.go Outdated

mitali-salvi added 3 commits June 10, 2026 15:48

Remove target_allocator TLS auto-injection

5d394bd

The opentelemetry.collect.prometheus path should not assume operator deployment model. If target_allocator needs TLS, users should specify cert paths in their prometheus config explicitly.

Simplify to plain Prometheus format only

51bd855

Remove OTel wrapper format detection and target_allocator TLS injection. Only standard Prometheus format (scrape_configs at root) is supported. No format ambiguity, no error string matching.

JayPolanco approved these changes Jun 10, 2026

View reviewed changes

okankoAMZ reviewed Jun 10, 2026

View reviewed changes

Address review feedback: sanitize config_path, use regex for cluster_…

b3881c2

…name - Add filepath.Clean on config_path before use - Replace character-by-character loop with regexp for cluster_name validation - Add comment explaining fwdConnector in both Exporters and Connectors

okankoAMZ approved these changes Jun 10, 2026

View reviewed changes

mitali-salvi merged commit 24dd5e0 into feature/opentelemetry Jun 11, 2026
360 of 387 checks passed

mitali-salvi deleted the feature/opentelemetry-prometheus branch June 11, 2026 13:30

Navigation Menu

Uh oh!

Add prometheus pipeline translator under opentelemetry config - #2148

Add prometheus pipeline translator under opentelemetry config#2148
mitali-salvi merged 7 commits into
feature/opentelemetryfrom
feature/opentelemetry-prometheus

mitali-salvi commented Jun 9, 2026 •

edited

Loading

Uh oh!

JayPolanco Jun 10, 2026

Uh oh!

mitali-salvi Jun 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

okankoAMZ Jun 9, 2026

Uh oh!

mitali-salvi Jun 10, 2026

Uh oh!

Uh oh!

Uh oh!

okankoAMZ Jun 10, 2026

Uh oh!

mitali-salvi Jun 10, 2026

Uh oh!

okankoAMZ Jun 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mitali-salvi commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Config

Behavior

Known limitation

Tests

Unit tests

EC2 integration test (us-west-2, mitsalvi-otel-test)

EKS integration test (us-east-1, cwa-otel-test)

Linting

Uh oh!

JayPolanco Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

mitali-salvi Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

okankoAMZ Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

mitali-salvi Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

okankoAMZ Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

mitali-salvi Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

okankoAMZ Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mitali-salvi commented Jun 9, 2026 •

edited

Loading

EC2 integration test (us-west-2, `mitsalvi-otel-test`)

EKS integration test (us-east-1, `cwa-otel-test`)