µ/log publisher for OpenTelemetry

[BrunoBonacci]

A few ppl have been asking an Open Telemetry compatible publisher for µ/log so I'm creating this ticket to track the discussion and the work.

The first thing to note is that we shouldn't need to piggyback on top of the java library (although it could be possible). If possible I'd like to use the same approach used for Zipkin which directly sends the data to the collector.

I'm including here a link to the OpenTelemetry specification for reference: https://opentelemetry.io/docs/reference/specification/

I'm curious about the contents of the discussion. Was anyone who asked wanting to migrate to it? Or is this greenfield? I considered otel, but between a core.async publisher and the provided prom, kafka publishers mulog handles all my needs. Is there a compelling reason for otel? I'm not seeing it in my use case.

Have you seen this lib?
clj-otel
It's got a library for the API, SDK, and several "extensions", like publishers in mu. It looks nice and all, but predictably doesn't have the same, just use it how you want approach mulog does.

I'm not sure how much you've looked into open-telemetry. Hopefully, more than I have. Open-telemetry is a far more prescriptive approach than mulog asks it's users to employ. It looks straight forward to write a publisher for log, but trace is much more involved. And there is predictably a fleshed out datamodel for everything.
log data model. It was enough to make me look at it all and say, don't tell me what to do DAD! It looks like a fat publisher. Not saying it isn't worth it.

Anyways, digging your libraries. I write these ultra-readable seq transformations using 'where' It's super cool.

[BrunoBonacci]

Thanks for the kind words.
I don't intend to use an open-telemetry library to implement the publisher. In my experience, the wrapping model is more prone to conflicts and unresolved bugs.
The approach I will use for this publisher is to make the HTTP request directly from the publisher like the zipkin publisher does at the moment.

[das-monki]

Is this still something you're working on?

Asking because I'd like to use datadog for tracing, and as far as I understand I have three options - 1) replace all uses of µ/trace with the equivalent of the datadog library, 2) use an adapter like this one that translates to open-telemetry style and send to datadog and 3) try out the auto-instrumentation of any jar provided by the datadog agent.

And btw, you're library has been my intro to structured logging, and has been a real pleasure using it, thank you for sharing! 😊

[BrunoBonacci]

Hi @das-monki,

thank you for your kind words.

yes is still my intention to work on an OpenTelemetry publisher for µ/log.
I think more realistically it will be around January time as I'm planning to work on a number of new features and changes over the holidays.

[steffan-westcott]

Hi, I'm the author of the clj-otel library mentioned in this discussion.

I think @BrunoBonacci suggestion of making a (OTLP?) publisher making HTTP calls directly to the OpenTelemetry Collector is the best way forward for mulog to integrate with OpenTelemetry. Using clj-otel would be an indirect approach and likely not be suitable. There is a significant overlap of mulog and Java OpenTelemetry (which clj-otel is based on) in terms of implementation, which suggests using them together in the same application would not be beneficial.

[margintop15px]

Hello @BrunoBonacci
Thank you for the awesome library and your work!
Is OpenTelemetry publisher a thing you're still working on? Any estimates when it can be usable?
Can I help somehow or contribute?
Having this publisher in hand will open so much possibilities to integrate very broad set of tools. Looking forward to see it in action

[nalcabio-tom]

The approach I will use for this publisher is to make the HTTP request directly

As someone who is trying to add OpenTelemetry to a Clojure app, I'm happy to hear this. Getting the Java sdk setup has been a total PITA.

[BrunoBonacci]

mulog-0.10.0 will contain a OpenTelemetry publisher which doesn't use java/clojure wrappers but it publishes the trace info directly to the a OpenTelemetry compatible collector via HTTP/JSON api.

In the next few days I will post a SNAPSHOT version, if anyone is interested in having early testing of this new publisher, please let me know via Slack.

https://clojurians.slack.com/archives/C03J782P329/p1734111950684399

[BrunoBonacci]

Hi All,

i’ve just published a SNAPSHOT version of the OpenTelemetry publisher.
[com.brunobonacci/mulog-opentelemetry "0.10.0-SNAPSHOT"]

it supports the publishing of logs and traces (metrics in mulog are just logs)
via the OpenTelemetry Protocol over HTTP/JSON.
It publishes the traces and logs directly without the use of wrappers and java libraries.

If you have the chance please give it try and send some feedbacks.
From what I can see the support for OpenTelemetry is quite scattered so make sure that your target system supports both logs and traces, alternatively just enable the publisher for the type of data it supports.

For more information see: https://cljdoc.org/d/com.brunobonacci/mulog/0.10.0-SNAPSHOT/doc/publishers/opentelemetry-publisher