Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions docs/configure-plugins/observability/openinference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ kind = "observability"
enabled = true

[components.config]
version = 1
version = 2

[components.config.openinference]
enabled = true
Expand Down Expand Up @@ -63,6 +63,7 @@ OpenInference uses the same OTLP section shape as
| `enabled` | `false` | Must be `true` to construct and register the subscriber. |
| `mark_projection` | `inherit` | `inherit` uses exporter-native handling; `event` forces span events; `tool` emits zero-duration OpenInference `TOOL` spans, parented as children when context is available. |
| `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `tool` projection; excluded marks retain exporter-native handling. Metadata `hook_event_name` aliases are also matched. |
| `attribute_mappings` | `[]` | Copies each fully qualified projected `key` to its `alias` without changing the OTLP type. Set both values to nonblank strings, and use each alias only once. |
| `transport` | `http_binary` | `http_binary` or `grpc`. |
| `endpoint` | Exporter default | OTLP endpoint. |
| `headers` | `{}` | String-to-string exporter headers. |
Expand Down Expand Up @@ -122,6 +123,38 @@ Cost appears as USD-denominated `llm.cost.total`. Refer to
[Token and Cost Field Semantics](/integrate-into-frameworks/provider-response-codecs#token-and-cost-field-semantics)
for the full mapping.

NeMo Relay projects top-level lifecycle payload fields to typed OTLP attributes
with dotted names. Non-LLM start metadata and all end metadata use the
`openinference.metadata` prefix, so `metadata = { tenant = "acme" }` becomes
`openinference.metadata.tenant = "acme"`. Non-LLM start events use
`nemo_relay.start.data`, `nemo_relay.start.input`, and
`nemo_relay.handle_attributes`. End events use `nemo_relay.end.data` and
`nemo_relay.end.output`. LLM start events use OpenInference semantic input
attributes instead of those generic start projections, and their final metadata
comes from the end event. Mark data, metadata, attributes, and category-profile
fields use the corresponding `nemo_relay.mark.*` prefixes.

Scalar strings, booleans, and numbers that fit an OTLP numeric type keep their
types. NeMo Relay emits larger unsigned integers as strings. When a top-level
field contains an object or array, NeMo Relay emits its value as a JSON string
at that field's dotted name. Nested `null` values remain in that string, but a
top-level field whose value is `null` is omitted. NeMo Relay no longer emits the
old aggregate `*_json` payload attributes. The exporter still keeps the
OpenInference `metadata` JSON-string attribute for backend compatibility.

Configure an alias when a backend expects a different attribute name:

```toml
[components.config.openinference]
attribute_mappings = [
{ key = "openinference.metadata.tenant", alias = "tenant.id" },
]
```

The source attribute remains in the span. If the span already contains
`tenant.id`, NeMo Relay keeps the existing value instead of replacing it with
the alias.

Redact sensitive event payloads with sanitize guardrails before production
export.

Expand Down Expand Up @@ -191,7 +224,7 @@ void (async () => {
version: 1,
components: [
observability.ComponentSpec({
version: 1,
version: 2,
openinference: observability.otlpConfig({
enabled: true,
transport: "http_binary",
Expand Down
37 changes: 35 additions & 2 deletions docs/configure-plugins/observability/opentelemetry.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ kind = "observability"
enabled = true

[components.config]
version = 1
version = 2

[components.config.opentelemetry]
enabled = true
Expand Down Expand Up @@ -57,6 +57,7 @@ The following table describes OpenTelemetry exporter settings:
| `enabled` | `false` | Must be `true` to construct and register the subscriber. |
| `mark_projection` | `inherit` | `inherit` uses exporter-native handling; `event` forces span events; `tool` emits zero-duration spans, parented as children when context is available, for trace-tree visibility. |
| `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `tool` projection; excluded marks retain exporter-native handling. Metadata `hook_event_name` aliases are also matched. |
| `attribute_mappings` | `[]` | Copies each fully qualified projected `key` to its `alias` without changing the OTLP type. Set both values to nonblank strings, and use each alias only once. |
| `transport` | `http_binary` | `http_binary` or `grpc`. |
| `endpoint` | Exporter default | OTLP endpoint. |
| `headers` | `{}` | String-to-string exporter headers. |
Expand Down Expand Up @@ -94,6 +95,38 @@ attributes. These values match ATIF `step.extra.ancestry.function_id` and
the trajectory-root span's `nemo_relay.uuid` also matches the ATIF `session_id`.
Backend-native `trace_id` and `span_id` values are not written into ATIF.

NeMo Relay projects top-level lifecycle payload fields to typed OTLP attributes
with dotted names. For example, `metadata = { tenant = "acme" }` becomes
`nemo_relay.start.metadata.tenant = "acme"`.
- Start events use the
`nemo_relay.start.data`, `nemo_relay.start.metadata`, and
`nemo_relay.start.input` prefixes.
- End events use `nemo_relay.end.data`,
`nemo_relay.end.metadata`, and `nemo_relay.end.output`.
- Handle attributes use
`nemo_relay.handle_attributes`. Mark data, metadata, attributes, and
category-profile fields use the corresponding `nemo_relay.mark.*` prefixes.
Comment thread
willkill07 marked this conversation as resolved.

Scalar strings, booleans, and numbers that fit an OTLP numeric type keep their
types. NeMo Relay emits larger unsigned integers as strings. When a top-level
field contains an object or array, NeMo Relay emits its value as a JSON string
at that field's dotted name. Nested `null` values remain in that string, but a
top-level field whose value is `null` is omitted. NeMo Relay no longer emits the
old aggregate `*_json` payload attributes.

Configure an alias when a backend expects a different attribute name:

```toml
[components.config.opentelemetry]
attribute_mappings = [
{ key = "nemo_relay.start.metadata.tenant", alias = "tenant.id" },
]
```

The source attribute remains in the span. If the span already contains
`tenant.id`, NeMo Relay keeps the existing value instead of replacing it with
the alias.

Coding-agent trace roots also carry these correlation attributes:

| Attribute | Meaning |
Expand Down Expand Up @@ -185,7 +218,7 @@ void (async () => {
version: 1,
components: [
observability.ComponentSpec({
version: 1,
version: 2,
opentelemetry: observability.otlpConfig({
enabled: true,
transport: "http_binary",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,22 @@ for example Arize Phoenix or another OpenInference-aware OTLP backend.

- OpenInference export is for OTLP backends that understand model-centric
OpenInference semantic conventions.
- Configure the exporter with the following settings:
- `transport`, `endpoint`, `service_name`, optional namespace and version,
instrumentation scope, headers, resource attributes, and timeout.
- `http_binary` transport and an OTLP/HTTP traces endpoint. Use `grpc` only
when a Tokio runtime is active.
- Exported events have the following mappings:
- Scope, tool, and LLM start inputs become `input.value`.
- Scope, tool, and LLM end outputs become `output.value`.
- Set `transport`, `endpoint`, and `service_name`, then add a namespace, version,
instrumentation scope, headers, resource attributes, timeout, or
`attribute_mappings` when needed.
- NeMo Relay projects lifecycle payload fields to typed OTLP attributes with
dotted names. Non-LLM start metadata and all end metadata use
`openinference.metadata`, while mark metadata uses
`nemo_relay.mark.metadata`.
- NeMo Relay emits a top-level object or array field as a JSON string, omits a
top-level `null` field, and no longer emits the old aggregate `*_json` payload
attributes.
- Use `attribute_mappings` to copy a fully qualified projected attribute to a
backend-specific alias without changing its OTLP type.
- Start with `http_binary` transport and an OTLP/HTTP traces endpoint. Use
`grpc` only when a Tokio runtime is active.
- Scope, tool, and LLM start inputs become `input.value`.
- Scope, tool, and LLM end outputs become `output.value`.
- LLM annotations follow the freshness rules:
- Each owning agent scope starts fresh, and a `compaction` mark refreshes it.
- The annotated input for the first subsequent LLM start retains complete
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,17 @@ OpenTelemetry Collector, Jaeger, Tempo, or Honeycomb.

- OpenTelemetry export maps NeMo Relay runtime events into OTLP traces for
tracing backends and collectors.
- Configure `transport`, `endpoint`, `service_name`, optional namespace and
version, instrumentation scope, headers, resource attributes, and timeout.
- Set `transport`, `endpoint`, and `service_name`, then add a namespace, version,
instrumentation scope, headers, resource attributes, timeout, or
`attribute_mappings` when needed.
- NeMo Relay projects start, end, handle, and mark payload fields to typed OTLP
attributes with dotted names. Start and end metadata use
`nemo_relay.start.metadata` and `nemo_relay.end.metadata`.
- NeMo Relay emits a top-level object or array field as a JSON string, omits a
top-level `null` field, and no longer emits the old aggregate `*_json` payload
attributes.
- Use `attribute_mappings` to copy a fully qualified projected attribute to a
backend-specific alias without changing its OTLP type.
- Start with `http_binary` transport and an OTLP traces endpoint such as a local
collector on port `4318` unless deployment requirements differ.
- `grpc` transport is available when a Tokio runtime is active.
Expand Down
Loading