Add opentelemetry root level config section with host_metrics, prometheus, OTLP, container_insights, and database_insights pipelines by mitali-salvi · Pull Request #2152 · aws/amazon-cloudwatch-agent

mitali-salvi · 2026-06-11T17:40:51Z

Summary

Adds the opentelemetry.collect config translation pipeline, enabling users to configure OTel pipelines via JSON config instead of raw YAML. This is the unified feature branch containing all v2 OTel collect sub-features.

Features

host_metrics — hostmetrics receiver (cpu, disk, filesystem, load, memory, network, processes) with configurable collection interval
container_insights — Full CI metrics pipeline (kubeletstats, cadvisor, node_exporter, dcgm, neuron, ebs-csi, lis-csi, kube-state-metrics, apiserver) with per-source processor chains, k8sattributes enrichment, and OTTL transforms
container_insights logs — Application and node log pipelines (filelog receiver) with dedicated exporters (compression: none for FluentBit migration parity)
prometheus — User-provided prometheus scrape config (config_path) with optional cluster_name enrichment
otlp — OTLP receivers (gRPC + HTTP) for metrics, logs, and traces with configurable endpoints
Shared metrics/logs/traces export pipelines — resourcedetection, batch, otlphttp export to CloudWatch OTLP endpoint with sigv4 auth and agenthealth
Identity and log routing processors — Platform-aware (host vs K8s) service.name, cloud.resource_id, deployment.environment derivation; log group/stream routing from aws.log.source

Architecture

Source Pipelines (per-feature):          Destination Pipelines (shared):
┌─────────────────────────┐             ┌──────────────────────────┐
│ host_metrics            │             │ metrics/opentelemetry    │
│ container_insights (x8) │──forward──▶│  resourcedetection       │
│ prometheus              │             │  identity transform      │
│ otlp (metrics)          │             │  batch → otlphttp        │
├─────────────────────────┤             ├──────────────────────────┤
│ otlp (logs)             │──forward──▶│ logs/opentelemetry       │
│ container_insights logs │             │  resourcedetection       │
├─────────────────────────┤             │  identity + routing      │
│ otlp (traces)           │──forward──▶│  batch → otlphttp        │
└─────────────────────────┘             └──────────────────────────┘

Key Design Decisions

rawMapConfig passthrough for CI templates — avoids factory struct round-trip bugs ($1 expansion, zero-value ErrorMode, filelog operator marshaling)
Per-source pipelines matching helm chart architecture (separate prometheus receivers, not consolidated)
forward connector fan-in to shared export pipelines
Explicit batch sizes from CW OTLP endpoint limits (1000 metrics, 10000 logs/traces per request; see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-OTLPEndpoint.html)
escapeDollarDigit scoped to CI translator only for k8sattributes regex backreferences

Testing

Full translator unit tests with golden YAML comparison
Schema validation tests
Verified on EKS cluster (cwa-otel-test) with all CI pipelines producing metrics
Lint passes (make lint clean)

…ing (#2135)

#2150)

…races exporters) Switch TestOtlpOtelConfig to use checkTranslation (like host_insights) so the TOML translator sets Global_Config.Region first, enabling the base export pipelines to generate. The golden YAML now includes otlphttp exporters, sigv4 auth, batch processors, and the complete pipeline routing.

…irectory Groups all opentelemetry.collect related golden files (host_insights, container_insights, dbi, prometheus, otlp) into a dedicated subdirectory for easier navigation.

…irectory

… files - Switch host_insights, prometheus, otlp, dbi to checkTranslation (full TOML+YAML path) so golden files include complete pipeline output - Keep checkTranslationNoValidation only for container_insights (K8s service account token validation fails in unit tests) but improved it to run TOML translation first so region/base pipelines generate - Regenerate all opentelemetry golden YAML files with full pipeline output - Remove unused mapstructure/otel imports where not needed

Aligns the logs pipeline with metrics and traces pipelines which both have resourcedetection for EC2/EKS metadata enrichment.

Previously only host_insights had a schema validation test. Now all opentelemetry.collect features have valid/invalid sample JSON files and corresponding test functions.

Aligns all three base pipelines with consistent agent health reporting. - Metrics: agenthealth/otlphttp_metrics (already had it) - Logs: agenthealth/logs (wraps headerssetter auth chain) - Traces: agenthealth/traces (wraps sigv4/xray auth)

…tion The function was only used in one place. Inlined the logic directly into TestContainerInsightsConfig with a comment explaining why collector validation is skipped (K8s service account token path is root-owned).

Add sample configs demonstrating both traditional (metrics/logs/traces) and opentelemetry.collect sections working together: - EC2 config: all v1 features + host_insights, database_insights, prometheus, otlp - EKS config: all v1 features + all v2 features including container_insights Includes golden files, translation tests, and schema validation.

…statement-level Pass the configured errorMode into buildStatements so statement-level error_mode matches the top-level value. Previously buildStatements hardcoded 'propagate' regardless of the WithErrorMode option. Remove unnecessary WithErrorMode("ignore") from otlp_log_source and logs_cleanup since their statements (set with where guard, delete_key) cannot produce errors and propagate is the correct default.

…2159)

…095f9

sky333999 · 2026-06-17T16:13:05Z

+	"github.com/aws/amazon-cloudwatch-agent/translator/translate/otel/common"
+	ci "github.com/aws/amazon-cloudwatch-agent/translator/translate/otel/pipeline/opentelemetry/containerinsights"
+	dbi "github.com/aws/amazon-cloudwatch-agent/translator/translate/otel/pipeline/opentelemetry/databaseinsights"
+	hi "github.com/aws/amazon-cloudwatch-agent/translator/translate/otel/pipeline/opentelemetry/hostmetrics"


nit: hi -> hm

sky333999 · 2026-06-17T16:16:11Z

@@ -0,0 +1,217 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.


nit: this file should be called translators.go to be consistent with the others.

sky333999 · 2026-06-17T21:16:49Z

+// CI logs pipelines are self-contained with dedicated exporters (compression: none)
+// to match the helm chart behavior for FluentBit migration parity. They cannot share
+// the base logs/opentelemetry exporter which uses gzip compression. See:
+// https://github.com/aws-observability/helm-charts/blob/main/charts/amazon-cloudwatch-observability/templates/linux/_otel-container-insights-config.tpl


nit: This reference in the helm chart is going to disappear from main pretty soon, either pin it to the commit or copy over the actual reasoning from the helm chart in here - like to prevent a cpu spike etc.

sky333999 · 2026-06-17T21:28:23Z

+//   - omitted: all pipelines
+func NewTranslators(conf *confmap.Conf) common.PipelineTranslatorMap {
+	translators := common.NewTranslatorMap[*common.ComponentTranslators, pipeline.ID]()
+	mode := getMode(conf)


As discussed, we should consider aligning this to the CollectionRole we have today. It will allow us to use the same default config in our helm and tweak the behavior with an env var.

We could continue to do this flag in addition to the env var and have a fallthrough ordering, but lets make the naming consistent i.e. instead of mode, might wanna call this role or something.

sky333999 · 2026-06-17T21:38:06Z

+	}
+
+	// Deployment metrics pipelines
+	if mode == "" || mode == "cluster" {


If mode is empty, itll end up doing the work for both cluster and node.. do we want that?
Might be safer to just do node level work if empty.

sky333999 · 2026-06-17T21:53:31Z

+		Region:             agent.Global_Config.Region,
+		CollectionInterval: collectionInterval.String(),
+		ScrapeTimeout:      scrapeTimeout.String(),
+		NodeName:           envOrPlaceholder("K8S_NODE_NAME"),


nit: While we set both, HOST_NAME is what we've historically used in the context of CI, so Id stick to that.

sky333999 · 2026-06-18T18:23:28Z

+		keys = make([]string, 0, len(sectionMap))
+		for k := range sectionMap {
+			keys = append(keys, k)
+		}


Suggested change

keys = make([]string, 0, len(sectionMap))

for k := range sectionMap {

keys = append(keys, k)

}

slices.Collect(maps.Keys(sectionMap))

sky333999 · 2026-06-18T21:24:15Z

+var scrapersWindowsConfig []byte
+
+// Config is a serializable representation of
+// hostmetricsreceiver.Config. The upstream type uses mapstructure:"-" on its


Can be a follow up but we should ideally update this in our fork instead of the below.

sky333999 · 2026-06-18T22:07:27Z

+
+var prometheusKey = common.ConfigKey(common.OpenTelemetryKey, common.CollectKey, common.PrometheusKey)
+var configPathKey = common.ConfigKey(prometheusKey, "config_path")
+var clusterNameKey = common.ConfigKey(prometheusKey, "cluster_name")


Can be a follow up, but some of these we should standerdize in this new schema. Like if cluster_name can be set here, why not for otlp.

sky333999 · 2026-06-18T22:36:11Z

+		"auth_type":   "serviceAccount",
+		"passthrough": false,
+		"filter": map[string]interface{}{
+			"node_from_env_var": "K8S_NODE_NAME",


related to the other comment if we do end up changing that to HOST_NAME at some point

sky333999 · 2026-06-18T23:07:33Z

+	MaxMetricsPerRequest = 1000
+	MaxLogsPerRequest    = 10000
+	MaxSpansPerRequest   = 10000
+	BatchTimeout         = 15 * time.Second


Should this be 15 or 30?

sky333999 · 2026-06-18T23:11:26Z

+}
+
+func isLocalhostEndpoint(endpoint string) bool {
+	return strings.HasPrefix(endpoint, "localhost") ||


nit: This seems a little fragile... consider parsing the host (net.SplitHostPort) and compare exactly.

sky333999 · 2026-06-18T23:12:34Z

+            "host_metrics": {
+              "type": "object",
+              "properties": {
+                "metrics_collection_interval": {


nit: We should probably standardize and call it just collection_interval

… node-only

musa-asad

.

JayPolanco and others added 11 commits May 28, 2026 16:31

Add shared OpenTelemetry pipeline framework (#2132)

c1c67a0

Add OpenTelemetry component translators for postgres database monitor…

60b480d

…ing (#2135)

Add pipeline translator for database monitoring (#2138)

922552e

Add host_insights pipeline to opentelemetry config section (#2136)

9bd1cb0

Add agenthealth metrics to opentelemetry shared pipeline (#2140)

2029328

Add OpenTelemetry container_insights pipeline translator (#2141)

51a9951

Move host_insights translator into its own package (#2145)

4fa0c9f

Add host insights to database monitoring (#2143)

1b502a7

Bump postgresqlreceiver version to filter monitoring user from top SQL (

28302b6

#2150)

Add prometheus pipeline translator under opentelemetry config (#2148)

24dd5e0

Add OTLP receiver pipeline and base traces pipeline (#2149)

01c5802

mitali-salvi requested a review from a team as a code owner June 11, 2026 17:40

Merge branch 'main' into feature/opentelemetry

37d2174

mitali-salvi added the ready for testing Indicates this PR is ready for integration tests to run label Jun 11, 2026

mitali-salvi changed the title ~~Feature/opentelemetry~~ Add opentelemetry.collect config section with host_insights, prometheus, OTLP, container_insights, and database_insights pipelines Jun 11, 2026

mitali-salvi added 2 commits June 11, 2026 21:24

Run go mod tidy to fix dependency resolution

1bf3ee0

mitali-salvi removed the ready for testing Indicates this PR is ready for integration tests to run label Jun 11, 2026

mitali-salvi added 11 commits June 11, 2026 21:41

Move opentelemetry sample configs to sampleConfig/opentelemetry/ subd…

576ac46

…irectory Groups all opentelemetry.collect related golden files (host_insights, container_insights, dbi, prometheus, otlp) into a dedicated subdirectory for easier navigation.

Move opentelemetry schema samples to sampleSchema/opentelemetry/ subd…

255bf95

…irectory

Add resourcedetection processor to base logs pipeline

b17442c

Aligns the logs pipeline with metrics and traces pipelines which both have resourcedetection for EC2/EKS metadata enrichment.

Add schema validation tests for database_insights, prometheus, and otlp

25edfe9

Previously only host_insights had a schema validation test. Now all opentelemetry.collect features have valid/invalid sample JSON files and corresponding test functions.

Add pipeline architecture documentation

7cc0c8b

Remove ARCHITECTURE.md

489d76b

Fix import ordering in tocwconfig_test.go

c329862

Rename host_insights config field to host_metrics (#2158)

d37c153

movence reviewed Jun 17, 2026

View reviewed changes

Comment thread translator/translate/otel/processor/transformprocessor/translate.go

Comment thread translator/config/schema.json

Comment thread translator/translate/otel/pipeline/opentelemetry/hostmetrics/translator.go

mitali-salvi and others added 5 commits June 17, 2026 16:08

Add solution and source scope attributes (#2160)

fc426b2

Merge branch 'main' into feature/opentelemetry

28e3c1c

Add k8sattributes processor to OTLP and Prometheus pipelines in K8s (#…

c2681ed

…2159)

Merge branch 'main' into feature/opentelemetry

62cf835

movence previously approved these changes Jun 18, 2026

View reviewed changes

jefchien and others added 4 commits June 18, 2026 16:44

Merge branch 'main' into feature/opentelemetry

a5d5bdb

Fix batch processor ID collision between metrics and traces pipelines

5a6398e

Add OTTL injection prevention for DBI username and instance_name

a5aad1a

Update amazon-contributing/opentelemetry-collector-contrib to 70530ca…

b9add7d

…095f9

mitali-salvi dismissed movence’s stale review via b9add7d June 18, 2026 20:59

jefchien reviewed Jun 18, 2026

View reviewed changes

Comment thread translator/tocwconfig/sampleConfig/opentelemetry/otlp_otel_config.yaml Outdated

mitali-salvi force-pushed the feature/opentelemetry branch from 23e5800 to 3213115 Compare June 18, 2026 21:45

Skip middleware for resourcedetection in opentelemetry pipelines

e547991

mitali-salvi force-pushed the feature/opentelemetry branch from 3213115 to e547991 Compare June 18, 2026 21:53

sky333999 reviewed Jun 18, 2026

View reviewed changes

Address review feedback: batch timeout 30s, CI empty mode defaults to…

bc0d2a6

… node-only

mitali-salvi force-pushed the feature/opentelemetry branch from 66280f8 to bc0d2a6 Compare June 18, 2026 23:42

sky333999 approved these changes Jun 18, 2026

View reviewed changes

jefchien approved these changes Jun 18, 2026

View reviewed changes

jefchien merged commit 460134e into main Jun 18, 2026
384 of 387 checks passed

jefchien deleted the feature/opentelemetry branch June 18, 2026 23:58

musa-asad reviewed Jun 19, 2026

View reviewed changes

JayPolanco mentioned this pull request Jul 14, 2026

[Fix] pin snowflake dependency to greater version to remove ophan processes #2202

Closed

		@@ -0,0 +1,217 @@
		// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Uh oh!

Conversation

mitali-salvi commented Jun 11, 2026 • edited by jefchien Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Features

Architecture

Key Design Decisions

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

musa-asad left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mitali-salvi commented Jun 11, 2026 •

edited by jefchien

Loading

musa-asad left a comment •

edited

Loading