[Observability] Issues with Consistency in Instrumentation

[AnoshanJ]

There are consistency issues in traces and spans, particularly as they relate to evaluation and trace visualisation.

Current State:

We are currently using a traceloop based instrumentation library(openllmetry). While this works functionally, we’ve identified several inconsistencies:

• Missing span information
• Some spans lack key attributes.
• Certain attribute fields are inconsistently populated.
• Inconsistent span naming
• Span names vary across:
  • Framework variations + versions
  • Framework-specific abstractions (e.g., LangGraph, nodes, tools)
  • Retrieval-related spans
  • Framework-specific divergence
  • Concepts like langgraph, nodes, and other higher-level abstractions introduce naming and structural inconsistencies.
  • Similar logical operations may appear differently depending on implementation details.

Why This Is a Problem:

These inconsistencies create friction in:

1. Evaluation Pipelines

• Hard to programmatically identify comparable spans.
• Difficult to aggregate metrics across runs.

2. Trace Attribute Visualization

• Attribute gaps cause incomplete dashboards.
• Naming inconsistencies make filtering unreliable.
• Harder to compare traces across experiments or versions.

3. Cross-Framework Comparability

• Retrieval spans may differ depending on integration.
• Node-level traces may not align semantically.