[receiver/datadogreceiver] Add EnableMultiTagParsing feature gate#44747
Merged
atoulme merged 3 commits intoopen-telemetry:mainfrom Dec 8, 2025
Merged
[receiver/datadogreceiver] Add EnableMultiTagParsing feature gate#44747atoulme merged 3 commits intoopen-telemetry:mainfrom
EnableMultiTagParsing feature gate#44747atoulme merged 3 commits intoopen-telemetry:mainfrom
Conversation
Datadog's metric model treats tags on data points as arbitrary strings. However, the product has special support for tags formatted as a `key:value` string. `key:value` tags allow slicing metrics in powerful ways and are core to many of Datadog's offerings. Since Datadog tags are arbitrary strings, there's nothing to stop users from recording tags with the same `key:` prefix on the same data point. In fact, Datadog's own integrations can do this: The `kube_service` tag from [the `kubelet` integration](https://docs.datadoghq.com/containers/kubernetes/tag/) will have one entry per Kubernetes service that fronts a pod. (An example set of tags might look something like `[...,"kube_service:datadog-cluster-agent-metrics-api","kube_service:datadog-cluster-agent-admission-controller",...]`). Currently, when the `datadogreceiver` receives multiple tags for a data point that share the same key, it only retains the last `key:value` pair as an OpenTelemetry attribe. In the above example, the attribute map would have an entry `kube_service = datadog-cluster-agent-admission-controller` - "datadog-cluster-agent-metrics-api" would have been discarded. This change introduces a feature gate that changes the tag parsing behaviour to support tags that share the same key. Instead of overwriting the previous `Str` entry in the attribute map, it converts the attribute map to a `Slice` and stores all values for that key. This conversion only happens if more than one value is encountered for the same key, so even with the feature flag enabled, there will not be any behaviour change for users who do not send tags that share the same key. I chose to implement this as a feature flag instead of a config setting because I believe this is technically correct behaviour, and should be the default in some future release of the OpenTelemetry Collector. Feedback on this decision is appreciated.
MovieStoreGuy
approved these changes
Dec 5, 2025
Contributor
|
This makes sense to me, and I don't see any issue with this approach. Thank you for putting it behind a feature gate to begin with :) |
atoulme
changed the title
EnableMultiTagParsing feature gateEnableMultiTagParsing feature gate
Contributor
|
Thank you for your contribution @goakley! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey. If you are getting started contributing, you can also join the CNCF Slack channel #opentelemetry-new-contributors to ask for guidance and get help. |
goakley
added a commit
to discord/opentelemetry-collector-contrib
that referenced
this pull request
Dec 9, 2025
…` feature gate The pair to open-telemetry#44747, this gets us on the way to converting OpenTelemetry attribute slices to individual Datadog tags. Currently when exporting to Datadog, attribute slices are converted to a "json-style" string, which is then turned into a valid Datadog tag. For example, the attribute `{kube_service = ["datadog-cluster-agent-admission-controller", "datadog-cluster-agent-metrics-api"]}` appears in Datadog as the tag `"kube_service:_datadog-cluster-agent-admission-controller_datadog-cluster-agent-metrics-api"` (the underscores are the result of converting the `[,]` json-style characters to [safe tag characters](https://docs.datadoghq.com/getting_started/tagging/#define-tags)). Following the same semantic convention as in open-telemetry#44747, slice attributes should instead be converted to individual tags. The above example would instead render in Datadog as the tags `"kube_service:datadog-cluster-agent-admission-controller","kube_service:datadog-cluster-agent-metrics-api"`. Datadog supports this sort of tag-key overlap on data points, and in fact uses it on purpose in [some of](https://docs.datadoghq.com/containers/kubernetes/tag/) its own integrations. This new behaviour is assumed to be the "correct" conversion behaviour, so it is implemented as a feature gate that will eventually become the default exporter behaviour (instead of being implemented as a config setting). Unfortuantely, there is an annoying circular dependency to deal with while implementing this. The `serializerexporter` package used by the exporter is part of [the `datadog-agent` repository](github.com/DataDog/datadog-agent/), but that package [depends on logic](https://github.com/DataDog/datadog-agent/blob/7.72.4/comp/otelcol/otlp/components/exporter/serializerexporter/factory.go#L22) in the `pkg/datadog` package of _this_ repository. To handle the cycle, the full change will be implemented as follows: 1. Add the feature gate for use in the `serializerexporter` package (this PR) 2. Update the logic in the `serializerexporter` package in the other repository 3. Update the logic in the `datadogexporter` package in this repository This creates a weird situation where the feature gate will do nothing for a period of time. I'm not sure if there's a good way to deal with this situation, other than to not document the feature gate until it's implemented.
goakley
added a commit
to discord/opentelemetry-collector-contrib
that referenced
this pull request
Dec 9, 2025
…pen-telemetry#44747) Datadog's metric model treats tags on data points as arbitrary strings. However, the product has special support for tags formatted as a `key:value` string. `key:value` tags allow slicing metrics in powerful ways and are core to many of Datadog's offerings. Since Datadog tags are arbitrary strings, there's nothing to stop users from recording tags with the same `key:` prefix on the same data point. In fact, Datadog's own integrations can do this: The `kube_service` tag from [the `kubelet` integration](https://docs.datadoghq.com/containers/kubernetes/tag/) will have one entry per Kubernetes service that fronts a pod. (An example set of tags might look something like `[...,"kube_service:datadog-cluster-agent-metrics-api","kube_service:datadog-cluster-agent-admission-controller",...]`). Currently, when the `datadogreceiver` receives multiple tags for a data point that share the same key, it only retains the last `key:value` pair as an OpenTelemetry attribe. In the above example, the attribute map would have an entry `kube_service = datadog-cluster-agent-admission-controller` - "datadog-cluster-agent-metrics-api" would have been discarded. This change introduces a feature gate that changes the tag parsing behaviour to support tags that share the same key. Instead of overwriting the previous `Str` entry in the attribute map, it converts the attribute map to a `Slice` and stores all values for that key. This conversion only happens if more than one value is encountered for the same key, so even with the feature flag enabled, there will not be any behaviour change for users who do not send tags that share the same key. I chose to implement this as a feature flag instead of a config setting because I believe this is technically correct behaviour, and should be the default in some future release of the OpenTelemetry Collector. Feedback on this decision is appreciated. Unit tests have been added against `tagsToAttributes` to make sure attribute conversion works as expected both with and without the feature gate. The collector has been built and run locally, confirming that exporters receive the expected (slice) attributes when multiple tags with the same key are received. The README for the receiver now describes how tags are processed, both with and without the feature gate.
goakley
added a commit
to discord/opentelemetry-collector-contrib
that referenced
this pull request
Dec 9, 2025
…` feature gate The pair to open-telemetry#44747, this gets us on the way to converting OpenTelemetry attribute slices to individual Datadog tags. Currently when exporting to Datadog, attribute slices are converted to a "json-style" string, which is then turned into a valid Datadog tag. For example, the attribute `{kube_service = ["datadog-cluster-agent-admission-controller", "datadog-cluster-agent-metrics-api"]}` appears in Datadog as the tag `"kube_service:_datadog-cluster-agent-admission-controller_datadog-cluster-agent-metrics-api"` (the underscores are the result of converting the `[,]` json-style characters to [safe tag characters](https://docs.datadoghq.com/getting_started/tagging/#define-tags)). Following the same semantic convention as in open-telemetry#44747, slice attributes should instead be converted to individual tags. The above example would instead render in Datadog as the tags `"kube_service:datadog-cluster-agent-admission-controller","kube_service:datadog-cluster-agent-metrics-api"`. Datadog supports this sort of tag-key overlap on data points, and in fact uses it on purpose in [some of](https://docs.datadoghq.com/containers/kubernetes/tag/) its own integrations. This new behaviour is assumed to be the "correct" conversion behaviour, so it is implemented as a feature gate that will eventually become the default exporter behaviour (instead of being implemented as a config setting). Unfortuantely, there is an annoying circular dependency to deal with while implementing this. The `serializerexporter` package used by the exporter is part of [the `datadog-agent` repository](github.com/DataDog/datadog-agent/), but that package [depends on logic](https://github.com/DataDog/datadog-agent/blob/7.72.4/comp/otelcol/otlp/components/exporter/serializerexporter/factory.go#L22) in the `pkg/datadog` package of _this_ repository. To handle the cycle, the full change will be implemented as follows: 1. Add the feature gate for use in the `serializerexporter` package (this PR) 2. Update the logic in the `serializerexporter` package in the other repository 3. Update the logic in the `datadogexporter` package in this repository This creates a weird situation where the feature gate will do nothing for a period of time. I'm not sure if there's a good way to deal with this situation, other than to not document the feature gate until it's implemented.
goakley
added a commit
to discord/opentelemetry-collector-contrib
that referenced
this pull request
Dec 9, 2025
…` feature gate The pair to open-telemetry#44747, this gets us on the way to converting OpenTelemetry attribute slices to individual Datadog tags. Currently when exporting to Datadog, attribute slices are converted to a "json-style" string, which is then turned into a valid Datadog tag. For example, the attribute `{kube_service = ["datadog-cluster-agent-admission-controller", "datadog-cluster-agent-metrics-api"]}` appears in Datadog as the tag `"kube_service:_datadog-cluster-agent-admission-controller_datadog-cluster-agent-metrics-api"` (the underscores are the result of converting the `[,]` json-style characters to [safe tag characters](https://docs.datadoghq.com/getting_started/tagging/#define-tags)). Following the same semantic convention as in open-telemetry#44747, slice attributes should instead be converted to individual tags. The above example would instead render in Datadog as the tags `"kube_service:datadog-cluster-agent-admission-controller","kube_service:datadog-cluster-agent-metrics-api"`. Datadog supports this sort of tag-key overlap on data points, and in fact uses it on purpose in [some of](https://docs.datadoghq.com/containers/kubernetes/tag/) its own integrations. This new behaviour is assumed to be the "correct" conversion behaviour, so it is implemented as a feature gate that will eventually become the default exporter behaviour (instead of being implemented as a config setting). Unfortuantely, there is an annoying circular dependency to deal with while implementing this. The `serializerexporter` package used by the exporter is part of [the `datadog-agent` repository](github.com/DataDog/datadog-agent/), but that package [depends on logic](https://github.com/DataDog/datadog-agent/blob/7.72.4/comp/otelcol/otlp/components/exporter/serializerexporter/factory.go#L22) in the `pkg/datadog` package of _this_ repository. To handle the cycle, the full change will be implemented as follows: 1. Add the feature gate for use in the `serializerexporter` package (this PR) 2. Update the logic in the `serializerexporter` package in the other repository 3. Update the logic in the `datadogexporter` package in this repository This creates a weird situation where the feature gate will do nothing for a period of time. I'm not sure if there's a good way to deal with this situation, other than to not document the feature gate until it's implemented.
songy23
pushed a commit
that referenced
this pull request
Dec 11, 2025
…` feature gate (#44859) #### Description The pair to #44747, this gets us on the way to converting OpenTelemetry attribute slices to individual Datadog tags. Currently when exporting to Datadog, attribute slices are converted to a "json-style" string, which is then turned into a valid Datadog tag. For example, the attribute `{kube_service = ["datadog-cluster-agent-admission-controller", "datadog-cluster-agent-metrics-api"]}` appears in Datadog as the tag `"kube_service:_datadog-cluster-agent-admission-controller_datadog-cluster-agent-metrics-api"` (the underscores are the result of converting the `[,]` json-style characters to [safe tag characters](https://docs.datadoghq.com/getting_started/tagging/#define-tags)). Following the same semantic convention as in #44747, slice attributes should instead be converted to individual tags. The above example would instead render in Datadog as the tags `"kube_service:datadog-cluster-agent-admission-controller","kube_service:datadog-cluster-agent-metrics-api"`. Datadog supports this sort of tag-key overlap on data points, and in fact uses it on purpose in [some of](https://docs.datadoghq.com/containers/kubernetes/tag/) its own integrations. This new behaviour is assumed to be the "correct" conversion behaviour, so it is implemented as a feature gate that will eventually become the default exporter behaviour (instead of being implemented as a config setting). Unfortunately, there is an annoying circular dependency to deal with while implementing this. The `serializerexporter` package used by the exporter is part of [the `datadog-agent` repository](https://github.com/DataDog/datadog-agent/), but that package [depends on logic](https://github.com/DataDog/datadog-agent/blob/7.72.4/comp/otelcol/otlp/components/exporter/serializerexporter/factory.go#L22) in the `pkg/datadog` package of _this_ repository. To handle the cycle, the full change will be implemented as follows: 1. Add the feature gate for use in the `serializerexporter` package (this PR) 2. Update the logic in the `serializerexporter` package in the other repository 3. Update the logic in the `datadogexporter` package in this repository This creates a weird situation where the feature gate will do nothing for a period of time. I'm not sure if there's a good way to deal with this situation, other than to not document the feature gate until it's implemented. #### Testing This PR contains none of the exporter logic (see above) so there is nothing to test. #### Documentation The feature flag is currently unimplemented (see above) so there is no new functionality to document.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Datadog's metric model treats tags on data points as arbitrary strings. However, the product has special support for tags formatted as a
key:valuestring.key:valuetags allow slicing metrics in powerful ways and are core to many of Datadog's offerings.Since Datadog tags are arbitrary strings, there's nothing to stop users from recording tags with the same
key:prefix on the same data point. In fact, Datadog's own integrations can do this: Thekube_servicetag from thekubeletintegration will have one entry per Kubernetes service that fronts a pod. (An example set of tags might look something like[...,"kube_service:datadog-cluster-agent-metrics-api","kube_service:datadog-cluster-agent-admission-controller",...]).Currently, when the
datadogreceiverreceives multiple tags for a data point that share the same key, it only retains the lastkey:valuepair as an OpenTelemetry attribe. In the above example, the attribute map would have an entrykube_service = datadog-cluster-agent-admission-controller- "datadog-cluster-agent-metrics-api" would have been discarded.This change introduces a feature gate that changes the tag parsing behaviour to support tags that share the same key. Instead of overwriting the previous
Strentry in the attribute map, it converts the attribute map to aSliceand stores all values for that key. This conversion only happens if more than one value is encountered for the same key, so even with the feature flag enabled, there will not be any behaviour change for users who do not send tags that share the same key.I chose to implement this as a feature flag instead of a config setting because I believe this is technically correct behaviour, and should be the default in some future release of the OpenTelemetry Collector. Feedback on this decision is appreciated.
Testing
Unit tests have been added against
tagsToAttributesto make sure attribute conversion works as expected both with and without the feature gate. The collector has been built and run locally, confirming that exporters receive the expected (slice) attributes when multiple tags with the same key are received.Documentation
The README for the receiver now describes how tags are processed, both with and without the feature gate.