[exporter/prometheusremotewrite] Support ottl in the header values

[eduardscaueru]

Component(s)

exporter/prometheusremotewrite

Is your feature request related to a problem? Please describe.

Currently, exporter headers in the OpenTelemetry Collector are static and cannot use dynamic or evaluated values. For example, if I want each remote write request to include a unique request-id header generated via UUID() or combined with other strings using Concat(), I have to hardcode the value or rely on external tooling.

Describe the solution you'd like

Use the existing OTTL (OpenTelemetry Transformation Language) ParseCondition method from opentelemetry-collector-contrib/pkg/ottl to enable dynamic evaluation of exporter header values.

Describe alternatives you've considered

No response

Additional context

Maybe under a feature flag for each header that enables ottl parsing and the default would be the current implementation as hardcoded values.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole @ArthurSens @ywwg

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Can you share more about your use-case?

Is there precedence for this kind of feature in other components? I'm concerned with how complex this will be to maintain, and it isn't a "normal" thing to do with PRW afaik.

[eduardscaueru]

Hello @dashpole

Sure. I have multiple apps that expose a Prometheus remote write like endpoint. You are right about the PRW, given the fact that is should not degrade the ingestion performance by adding headers and logs to the request, but since the PRW spec can be reproduced in multiple coding languages (Go, Java etc.), other scalable services can be built on it.

Tracing in a microservice architecture, that uses these kind of PRW services, is a bit trickier, but I guess achievable by propagating from the source (which is the otel collector in my case) a request identifier.

[dashpole]

Are you specifically trying to add trace instrumentation to the exporter? If so, we can work on that as a feature without adding generic OTTL-based header support.

[eduardscaueru]

If the trace instrumentation on the exporter would log and include the trace context, according to W3C, in the HTTP headers to the downstream services, then I think the services can be linked.

[dashpole]

Yes, it would do context propagation

[eduardscaueru]

Should I repurpose the issue or create another one?

Also, is there any chance of me and my friend implementing this feature?

[axw]

FWIW, we needed to do something sort of similar for a couple of other exporters (kafka, elasticsearch). What we ended up doing was adding an include_metadata_keys config to propagate headers from the request context to the destination.

If you need to set dynamic headers, that can be done with a different component in the middle. e.g. see #39157

[eduardscaueru]

Just to make sure I understand, in order to achieve this, it would be needed to:

1. Add a ottl processor with a UUID in some context
2. Change the exporter code to get the request id from this context and put it in the headers