Preserve custom mark events in ATIF and visual trace representations

[bbednarski9]

Summary

NeMo Relay emits custom mark events to subscribers and telemetry exporters, but downstream trajectory and trace views do not consistently preserve them as visible, semantically meaningful events.

The event data is present in the raw event stream and can be present in OTLP span events, yet generic ATIF consumers may only see a system step with extra.event_payload, while native ATIF import and visual trace views do not expose the mark as an event or step that users can easily inspect.

This makes it difficult to verify event ordering, parentage, and payloads for integrations that rely on custom marks.

Observed behavior

• A mark with a name, payload, timestamp, and parent scope is emitted by Relay.
• ATOF preserves the mark as a raw event.
• Generic ATIF serialization preserves the payload and ancestry, but represents the mark as a system step without an observation or tool/event representation.
• Native ATIF import into downstream trace tooling may accept the trajectory while omitting the custom mark from the visible trace structure.
• OTLP/OpenInference exporters attach a mark to an active parent span as a span event; this preserves the data, but it may not appear as a distinct node in visual trace trees.
• The result differs depending on the export format and downstream consumer, despite originating from the same Relay event.

Expected behavior

Relay should provide a generic, documented representation for custom mark events that preserves:

• mark name and category/profile;
• payload and metadata;
• event timestamp;
• parent UUID and ancestry;
• ordering relative to surrounding scope, tool, and LLM events.

The representation should remain generic and must not depend on any particular plugin or mark name. It should also preserve the distinction between a point-in-time mark and a scoped tool or LLM operation; marks should not be silently converted into synthetic tool calls unless that is an explicit consumer projection.

Possible directions

• Define a generic ATIF mark-step convention that downstream consumers can render directly.
• Add an exporter/renderer adapter mechanism for mark categories without hard-coding product-specific names in Relay core.
• Update the native ATIF import path or downstream integration guidance to preserve custom mark steps as visible events.
• Ensure OTLP/OpenInference consumers can inspect span events and retain the same parentage and ordering guarantees.

Acceptance criteria

• Add a neutral fixture mark with structured data and a parent scope.
• Verify that ATOF, ATIF, OTLP, and OpenInference retain the mark name, payload, timestamp, parentage, and ordering.
• Verify that native ATIF import exposes the mark as a discoverable event/step rather than silently reducing it to opaque metadata.
• Verify that visual trace consumers can inspect the mark without requiring product-specific knowledge.
• Add regression coverage for marks emitted before and after parent scope completion, including the orphan-mark fallback behavior.
• Do not introduce private marker fields or product-specific special cases into the generic Relay API.