net_http with request headers as span attributes

[isaiahshiner]

My actual problem

I have a legacy X-MyCompany-RequestId header that is sent with every request and relayed throughout the backend. I'd like to log this in spans, so that you can connect OTEL spans with our legacy observability stuff. This will be replaced with propagator, but not everything is on OTEL yet.

This affects multiple instrumentations, but net_http is the main one. I'm currently only using standard Rails instrumentation from this repo, I don't have any custom instrumentation/spans. It seems like I could wrap every call to Net::HTTP with a new span, but that seems like a bad approach.

Questions and workflow

1. How should HTTP Headers be reported? I found semantic docs, but not recommendations. Dumping all headers does seem like a bad idea.
   a. Wait, found it, its under footnote [15] here.

   [15] http.request.header.: Instrumentations SHOULD require an explicit configuration of which headers are to be captured. Including all request headers can be a security risk - explicit configuration helps avoid leaking sensitive information.

   b. Yep, this makes sense. So, my expected workflow would be to add X-MyCompany-RequestId to the expected headers of Net:HTTP during initialization?

2. Is this just unfinished work? Is it planned? It seems approachable enough that I might be able to pick it up.

[thompson-tomo]

So, my expected workflow would be to add X-MyCompany-RequestId to the expected headers of Net:HTTP during initialization?

Yes you would be expected to define which request & response headers to capture. Note there is an effort (Declarative Config) to define this in a consistent manner across languages.

[robbkidd]

Hi @isaiahshiner! Configuration is a bit of a puzzle at the moment until we get to the promised land of Declarative Config. My apologies that the documentation for instrumentation options are lacking (and sometimes non-existent!). I expect as the project starts in on Declarative Config (timing to-be-determined), options will become more consistently implemented across the instrumentation. Thanks for opening this issue!

I have a legacy header that is sent with every request and relayed throughout the backend. I'd like to log this in spans …
I'm currently only using standard Rails instrumentation …

Server-side: You've got an option. Using the Rails instrumentation means you are using the Rack instrumentation under the hood. The rack instrumentation has an :allowed_request_headers option for capturing a list of headers on incoming requests to be recorded on the server spans it produces.

I could wrap every call to Net::HTTP with a new span …

Client-side: No solution today. You are correct that the Net::HTTP instrumentation does not (yet?) have an option to provide an allow list of request headers to be recorded on the client spans it produces.

Is it important for your case that you have this header also recorded on client spans?

[isaiahshiner]

Ah okay, I'll give the rack instrumentation config a try! Thanks for the reply.

All these spans should be serverside, but the rails here is a middleware, making requests to various actual backend services using Net::HTTP. I'm not sure whether rack gets in on those. I was focused on instrumenting this from Net::HTTP because that's what the spans show up as in the trace client, and I think that's where the requestId is actually added. But I suppose rack might be involved one step up?

[arielvalentin]

How do you propagate your custom headers?

Are they available as thread local variables?

[isaiahshiner]

How do you propagate your custom headers? Are they available as thread local variables?

Yes? Honestly, I don't know the Ruby well enough to answer this, but seems like its either set on Thread.current[:request_id] or env['HTTP_X_MYCOMPANY_REQUESTID'], but I'm not sure how those values get set. There's also a lot of inconsistency in how requests are actually made. There's even some httparty stuff I'm not even trying to handle right now.

So far, this has been a small project relying on standard instrumentation, and that's why I started to ask how I can get the instrumentation to use the requestId it implicitly has.

[arielvalentin]

If they are fiber local variables you can write your own span processor that sets your attribute to all spans on_start, check out the Baggage Processor in the contrib repo as a model of how to do that