Skip to content

[propagator-xray] X-Amzn-Trace-Id inject key is mixed-case, breaking gRPC metadata usage #2129

@gonsuke

Description

@gonsuke

Description of the bug

When OpenTelemetry::Propagator::XRay::TextMapPropagator is used as a propagator and OpenTelemetry.propagation.inject(metadata) is called with a gRPC metadata hash, the key X-Amzn-Trace-Id (mixed-case) is injected. However, gRPC requires metadata keys to match [a-z0-9-_.] (lowercase only), this causes a runtime error.

The relevant line in text_map_propagator.rb

XRAY_CONTEXT_KEY = 'X-Amzn-Trace-Id'  # line 20
# ...
setter.set(carrier, XRAY_CONTEXT_KEY, xray_value)  # line 98

Share details about your runtime

  • Operating system details: Linux Debian bookworm, macOS 26.3.1
  • Ruby 3.4.4
  • opentelemetry-propagator-xray: 0.26.1
  • opentelemetry-sdk: 1.10.0
  • grpc gem (Ruby gRPC)

Share a simplified reproduction if possible

require 'opentelemetry/sdk'
require 'opentelemetry-propagator-xray'
require 'grpc'

OpenTelemetry::SDK.configure do |c|
  c.propagators = [
    OpenTelemetry::Trace::Propagation::TraceContext.text_map_propagator,
    OpenTelemetry::Propagator::XRay::TextMapPropagator.new
  ]
  c.service_name = 'repro'
end

metadata = {}
tracer = OpenTelemetry.tracer_provider.tracer('repro')
tracer.in_span('test') do
  OpenTelemetry.propagation.inject(metadata)
end

puts "Injected keys : #{metadata.keys.inspect}"

begin
  chan = GRPC::Core::Channel.new('localhost:9999', nil, :this_channel_is_insecure)
  call = chan.create_call(nil, nil, '/blah/method', nil, Time.now + 1)
  call.run_batch(GRPC::Core::CallOps::SEND_INITIAL_METADATA => metadata)
rescue GRPC::BadStatus, GRPC::Core::CallError => e
  puts "gRPC error     : #{e.class}: #{e.message}"
rescue StandardError => e
  puts "Other error    : #{e.class}: #{e.message}"
end

Output:

Injected keys : ["traceparent", "X-Amzn-Trace-Id"]
Other error    : ArgumentError: 'X-Amzn-Trace-Id' is an invalid header key, must match [a-z0-9-_.]+

Workaround

I worked around this by wrapping the gRPC metadata hash in a carrier that normalizes keys to lowercase before writing:

class LowercaseCarrier
  def initialize(hash) = @hash = hash
  def []=(key, value)  = @hash[key.downcase] = value
  def [](key)          = @hash[key.downcase]
  def keys             = @hash.keys
end

OpenTelemetry.propagation.inject(LowercaseCarrier.new(metadata))

The AWS X-Ray header name is case-insensitive at the HTTP level (per RFC 7230), so I'd imagine lowercase would be fine... but I wanted to double check here before assuming anything.

Thanks for maintaining these gems. It's been a solid building block for our tracing setup on AWS.

Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more in our end user docs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions