Skip to content
29 changes: 19 additions & 10 deletions .agents/skills/add-middleware/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,26 +24,31 @@ middleware behavior to a new pipeline stage.

Decide these before editing code:

- Is this for tools, LLMs, or both?
- Is this for tools, LLMs, marks, scope events, or a combination?
- Is it a conditional guardrail, sanitize guardrail, request intercept, or
execution intercept?
- Does it run on request input, inner callable execution, stream chunks, or
final response output?
- Is the callback fallible, and how should callback failures propagate?
- Does it need both global and scope-local registration?
- What should subscribers observe in `event.input` and `event.output` after this
- What should subscribers and exporters observe in the event payload after this
middleware runs?
- If this is an event sanitizer, which of `data`, `category_profile`, and
`metadata` can change, and is the event used only as immutable context?

## Pipeline Order

See `docs/about/concepts/middleware.md` for the full diagrams.
Refer to `docs/about-nemo-relay/concepts/middleware.mdx` for the full diagrams.

- **Tool execute**:
conditional guardrails -> request intercepts -> sanitize request (for events)
| execution intercept chain(callable) -> sanitize response
- **LLM execute**:
conditional guardrails -> request intercepts -> sanitize request (for events)
| execution intercept chain(callable) -> sanitize response
- **Mark and scope events**:
specialized tool or LLM sanitizer (when applicable) -> mark or scope event
sanitizer -> subscriber and exporter dispatch

## Core Steps

Expand Down Expand Up @@ -73,8 +78,11 @@ Follow the pattern of `tool_sanitize_request_chain` or `tool_request_intercepts_

5. Wire the chain into the execute path.

Update `crates/core/src/api/tool.rs` or `crates/core/src/api/llm.rs` to call
the new chain method at the appropriate pipeline stage.
Update the relevant lifecycle owner to call the new chain method at the
appropriate pipeline stage. Tool and LLM paths live in
`crates/core/src/api/tool.rs` and `crates/core/src/api/llm.rs`; shared mark and
scope event sanitization lives in `crates/core/src/api/shared.rs` and is called
from `crates/core/src/api/scope.rs`.

6. Expose the new middleware surface in every affected binding.

Expand All @@ -85,9 +93,10 @@ Follow the `add-binding-feature` skill for the cross-binding implementation chec
- [ ] Registration and duplicate-name behavior
- [ ] Deregistration and no-op missing-name behavior
- [ ] Ordering by priority
- [ ] Callback error propagation
- [ ] Callback failure policy, including fail-open behavior when required
- [ ] Scope-local registration, inheritance, and cleanup on pop
- [ ] Event input/output semantics after middleware mutation
- [ ] Event payload semantics after middleware mutation
- [ ] Mark and scope event field semantics, including immutable identity fields
- [ ] Parity coverage in every affected binding

## Key References
Expand All @@ -97,7 +106,7 @@ Follow the `add-binding-feature` skill for the cross-binding implementation chec
- Runtime state and chain builders: `crates/core/src/api/runtime/state.rs`
- Scope-local registry merging: `crates/core/src/context/registries.rs`
- Registry: `crates/core/src/registry.rs`
- Pipeline docs: `docs/about/concepts/middleware.md`
- Architecture docs: `docs/about/architecture.md`
- Registration examples: `docs/instrument-applications/advanced-guide.md`
- Pipeline docs: `docs/about-nemo-relay/concepts/middleware.mdx`
- Architecture docs: `docs/about-nemo-relay/architecture.mdx`
- Registration examples: `docs/instrument-applications/advanced-guide.mdx`
- Validation: `validate-change`
8 changes: 7 additions & 1 deletion crates/node/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ should install it from npm rather than depend on the Rust crate directly.

## Why Use It?

Use the Node.js binding for the following tasks:

- **Own execution context in Node.js**: Group agent, tool, and LLM work into
one scope tree from JavaScript or TypeScript.
- **Put policy around callbacks**: Register guardrails and intercepts for
Expand All @@ -37,11 +39,15 @@ should install it from npm rather than depend on the Rust crate directly.

## What You Get

The Node.js package provides the following capabilities:

- **npm package for Node.js**: A Node.js 24 or newer package backed by a
napi-rs native extension.
- **Managed tool and LLM execution**: Helpers that emit lifecycle events and
run middleware in a consistent order.
- **Middleware APIs**: Guardrails and intercepts for tool and LLM boundaries.
- **Middleware APIs**: Guardrails and intercepts for tool and LLM boundaries,
plus mark and scope event sanitizers for `data`, `categoryProfile`, and
`metadata`.
- **Observability exporters**: Subscriber and exporter support for common
runtime telemetry flows.
- **Additional entry points**: `nemo-relay-node/typed`,
Expand Down
6 changes: 6 additions & 0 deletions docs/about-nemo-relay/concepts/events.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,12 @@ exporters. Consumers should prefer typed event fields, binding helpers, and
annotated request or response data before falling back to raw payload
re-reading.

Before that handoff, event sanitizers can replace `data`, `category_profile`,
and `metadata`. The runtime keeps identity, lifecycle, category, timestamp,
and parent fields immutable. Refer to
[Event Sanitizers](/reference/event-sanitizers) for registration levels and
ordering.

Events preserve runtime facts. They do not decide which scope owns a call, how
replay or cost policy is applied, how redaction policy is configured, how
streams are blocked, or how exporter-specific semantic projection works. Those
Expand Down
72 changes: 52 additions & 20 deletions docs/about-nemo-relay/concepts/middleware.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ import { MermaidStyles } from "@/components/MermaidStyles";
{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0 */}

This page explains the runtime behavior that runs around managed tool and LLM calls.
This page explains the runtime behavior that runs around managed tool and LLM
calls and sanitizes emitted mark and scope events.

## What Middleware Is

Middleware is the runtime behavior that runs around tool and LLM execution.
NeMo Relay uses middleware to control, transform, or observe work at specific
lifecycle points.
Middleware controls or transforms tool and LLM execution and sanitizes emitted
events. NeMo Relay applies each surface at a specific lifecycle point.

Middleware is organized by lifecycle meaning rather than as one undifferentiated
hook system.
Expand Down Expand Up @@ -106,6 +106,22 @@ Sanitize-response guardrails rewrite the payload recorded on emitted end events.

Use them when the event stream should hide or reduce sensitive response data.

### Sanitize Mark and Scope Events

Event sanitizers cover observability fields that are outside the specialized
tool and LLM payload APIs. Separate registries apply to marks, scope starts,
and scope ends. They can rewrite `data`, `category_profile`, and `metadata`
while receiving the complete event as immutable context.

Scope event sanitizers run for every category. On tool and LLM scope events,
they run after the specialized request or response sanitizer. Mark sanitizers
cover explicit marks and marks materialized by middleware, plugins, and
streaming lifecycle helpers.

Register event sanitizers globally, on an owning scope, or through a plugin
context. For the callback contract and binding APIs, refer to
[Event Sanitizers](/reference/event-sanitizers).

Sanitize guardrails are observability-oriented. They do not rewrite the real
arguments passed to the callback or the real value returned to the caller.

Expand All @@ -123,44 +139,51 @@ sequenceDiagram
participant Req as Request Intercepts
participant Exec as Execution Intercepts
participant Callback as Real Callback
participant San as Sanitize Guardrails
participant San as Tool / LLM Sanitizers
participant EventSan as Mark / Scope Event Sanitizers
participant Dispatch as Async Subscriber Dispatcher
participant Subs as Subscribers
participant Consumers as Subscribers / Exporters

Caller->>Runtime: managed tool or LLM call
Runtime->>Cond: decide whether work may proceed

alt blocked
Runtime->>EventSan: sanitize guardrail scopes and rejection mark
Cond-->>Caller: reject execution
else allowed
Runtime->>Req: rewrite the real request
Runtime->>San: sanitize emitted start payload
Runtime->>EventSan: sanitize start event fields
Runtime->>Dispatch: enqueue start event before execution
Dispatch-->>Subs: deliver start event later
Dispatch-->>Consumers: deliver start event later
Runtime->>Exec: wrap execution
Exec->>Callback: invoke callback
Callback-->>Exec: return real result
Exec-->>Runtime: continue
Runtime->>San: sanitize emitted end payload
Runtime->>EventSan: sanitize end event fields
Runtime->>Dispatch: enqueue end event
Dispatch-->>Subs: deliver end event later
Dispatch-->>Consumers: deliver end event later
Runtime-->>Caller: return real result
end
```

1. Conditional-execution guardrails
2. Request intercepts
3. Sanitize-request guardrails and emit the start event
4. Execution intercepts
5. The real callback, unless an execution intercept replaces it
6. Sanitize-response guardrails and emit the end event
3. Tool or LLM sanitize-request guardrails
4. Scope-start event sanitizers and start-event emission
5. Execution intercepts
6. The real callback, unless an execution intercept replaces it
7. Tool or LLM sanitize-response guardrails
8. Scope-end event sanitizers and end-event emission

For streaming LLM flows, the same pre-execution order applies: the runtime
applies `sanitize-request` guardrails and emits the LLM start event before the
stream execution intercept chain runs. Stream execution intercepts are the
execution family for streaming provider callbacks. The runtime then collects
chunks and finalizes the stream before `sanitize-response` guardrails rewrite
the emitted end-event payload at item 6.
the emitted end-event payload and scope-end event sanitizers run at items 7 and
8.

This ordering is what makes the semantic split between intercepts and
guardrails important:
Expand Down Expand Up @@ -204,23 +227,28 @@ flowchart TB
direction TB
SanitizeRequestGuardrails[/Sanitize Request Guardrail/]
SanitizeResponseGuardrails[/Sanitize Response Guardrail/]
MarkSanitizers[/Mark Event Sanitizers/]
ScopeStartSanitizers[/Scope-Start Event Sanitizers/]
ScopeEndSanitizers[/Scope-End Event Sanitizers/]
StartEvent[Emit Start Event]
EndEvent[Emit End Event]
Dispatcher[["Async Subscriber Dispatcher"]]
EventSubscribers[["Event Subscribers"]]
EventConsumers[["Subscribers / Exporters"]]
end
end

Response([Response])

Request --> ConditionalExecutionGuardrails
RequestIntercepts -->|Transformed Request| SanitizeRequestGuardrails
ConditionalExecutionGuardrails -->|"(rejected event)"| Dispatcher
ConditionalExecutionGuardrails -->|"(rejection mark)"| MarkSanitizers
MarkSanitizers -->|Sanitized Mark Fields| Dispatcher
ConditionalExecutionGuardrails -->|"(rejected)"| RaiseException
ConditionalExecutionGuardrails -->|"(passed)"| RequestIntercepts
SanitizeRequestGuardrails -->|Sanitized Start Payload| StartEvent
SanitizeRequestGuardrails -->|Sanitized Start Payload| ScopeStartSanitizers
ScopeStartSanitizers -->|Sanitized Event Fields| StartEvent
StartEvent --> Dispatcher
Dispatcher --> EventSubscribers
Dispatcher --> EventConsumers
StartEvent -->|Before Execution Intercepts| HasExecutionIntercept
RequestIntercepts -.->|Real Request| HasExecutionIntercept

Expand All @@ -239,13 +267,14 @@ flowchart TB
Finalizer -.->|Aggregated Response| SanitizeResponseGuardrails
Finalizer o--o|shared state| Collector

SanitizeResponseGuardrails -->|Sanitized End Payload| EndEvent
SanitizeResponseGuardrails -->|Sanitized End Payload| ScopeEndSanitizers
ScopeEndSanitizers -->|Sanitized Event Fields| EndEvent
EndEvent --> Dispatcher

class Execution,Invocation,Streaming,Observability,Request,Response grey-lightest;
class Dispatcher,EventSubscribers,StartEvent,EndEvent teal-lightest;
class Dispatcher,EventConsumers,StartEvent,EndEvent teal-lightest;
class RequestIntercepts,HasExecutionIntercept,ExecutionIntercepts yellow-lightest;
class ConditionalExecutionGuardrails,SanitizeRequestGuardrails,SanitizeResponseGuardrails green-lightest;
class ConditionalExecutionGuardrails,SanitizeRequestGuardrails,SanitizeResponseGuardrails,MarkSanitizers,ScopeStartSanitizers,ScopeEndSanitizers green-lightest;
class RaiseException red-lightest;
class DefaultCallable,InterceptResult,Collector,Finalizer magenta-lightest;
```
Expand All @@ -261,6 +290,9 @@ Use these comparisons to pick the middleware surface that matches the behavior y
boundary.
- Use a **sanitize guardrail** when only subscribers and exporters should see
rewritten data.
- Use a **mark or scope event sanitizer** when the sensitive fields are in
`data`, `category_profile`, or `metadata` rather than the managed tool or LLM
request/response payload.
- Use a **stream execution intercept** when you need streaming-specific
behavior applied across the lifecycle of a long-lived or chunked response,
such as per-chunk transformation, incremental authorization, logging or
Expand Down
2 changes: 2 additions & 0 deletions docs/build-plugins/language-binding/about.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ references during registration.
A plugin can install one or more of these runtime surfaces:

- **Subscribers:** Event subscribers.
- **Event sanitizers:** Mark, scope-start, and scope-end observability-field
sanitizers.
- **Tool middleware:** Sanitize-request and sanitize-response guardrails,
conditional-execution guardrails, request intercepts, and execution
intercepts.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,8 @@ Prefer observe-only defaults for new policies and execution-affecting intercepts
Use `PluginContext` to register:

- Subscribers
- Mark event sanitizers
- Scope-start and scope-end event sanitizers
- Tool guardrails
- Tool request and execution intercepts
- LLM guardrails
Expand Down
8 changes: 8 additions & 0 deletions docs/build-plugins/language-binding/register-behavior.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@ Use the context only after validation succeeds. Keep validation deterministic an
side-effect free. Inspect configuration and return diagnostics. Create runtime
objects and attach them to the context during registration.

The context includes mark, scope-start, and scope-end event sanitizer
registrations in addition to the tool and LLM middleware surfaces. Event
sanitizer callbacks receive the immutable event plus `data`,
`category_profile`, and `metadata`, and return only those observability fields.
Use the context methods so component name qualification and rollback also apply
to these registries. Refer to [Event Sanitizers](/reference/event-sanitizers)
for the binding-specific method names.

## Header Plugin Example

The same model applies in every binding: validate component-local config, then install middleware through the component-scoped registration context.
Expand Down
15 changes: 11 additions & 4 deletions docs/configure-plugins/pii-redaction/about.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ position: 1
SPDX-License-Identifier: Apache-2.0 */}

Use the PII redaction plugin when you want first-party privacy controls for
managed NeMo Relay LLM and tool observability surfaces through the shared
plugin system.
managed NeMo Relay LLM, tool, mark, and generic scope observability surfaces
through the shared plugin system.

The built-in plugin component uses `kind = "pii_redaction"` and is available as
a first-party NeMo Relay plugin.
Expand All @@ -30,7 +30,8 @@ The plugin supports these backend modes:

Start here when you need to:

- Remove sensitive fields from emitted tool or LLM payloads.
- Remove sensitive fields from emitted tool, LLM, mark, or generic scope
payloads.
- Replace sensitive text with a deterministic marker such as `[REDACTED]`.
- Hash matching values before observability exporters or subscribers receive
them.
Expand Down Expand Up @@ -66,10 +67,11 @@ The difference between middleware and `pii_redaction` is that middleware is the

## Current Scope

The built-in plugin currently exposes four managed sanitize surfaces:
The built-in plugin exposes five managed sanitize surfaces:

- `input`
- `output`
- `mark`
- `tool_input`
- `tool_output`

Expand All @@ -88,6 +90,11 @@ The current backend boundary is intentional:
- Managed LLM surfaces use the selected built-in codec so redaction can target
normalized Relay request and response shapes such as `/messages/0/content`
and `/message`.
- `mark` sanitizes `data`, `category_profile`, and `metadata` independently on
every mark event. It defaults to `true`; set `mark = false` to opt out.
- `input` and `output` also sanitize those three fields on non-tool, non-LLM
scope starts and ends. Tool and LLM scope envelopes remain on their
specialized sanitizer paths to avoid applying the same mask or hash twice.

## Observability Boundary

Expand Down
Loading
Loading