Skip to content

feat(translation): harden cross-protocol translation#46

Merged
bbednarski9 merged 2 commits into
NVIDIA-NeMo:topic/nemo-relay-integrationfrom
bbednarski9:feature/relay-translation-library
Jul 9, 2026
Merged

feat(translation): harden cross-protocol translation#46
bbednarski9 merged 2 commits into
NVIDIA-NeMo:topic/nemo-relay-integrationfrom
bbednarski9:feature/relay-translation-library

Conversation

@bbednarski9

@bbednarski9 bbednarski9 commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown

What

Harden the public switchyard-translation crate for downstream Rust consumers that need to make Switchyard routing decisions actionable across provider protocols.

  • Normalize Anthropic base64 image sources and OpenAI data URIs through the shared image representation.
  • Preserve mixed text and tool-call outputs when translating OpenAI Responses results to OpenAI Chat Completions or Anthropic Messages.
  • Parse Responses function-call arguments before creating normalized tool calls.
  • Propagate OpenAI Chat and Responses stream failures without synthesizing a later success terminal.
  • Preserve response.incomplete for Responses clients and map its reason to valid Chat or Anthropic terminal semantics.
  • Declare Rust 1.93 as the crate-specific minimum supported Rust version so the library can be consumed by the Relay plugin without raising Relay's toolchain requirement.
  • Add focused request, buffered-response, and streaming regression coverage.

Why

Switchyard should own the provider-protocol translation engine needed to execute its routing decisions. A reusable library lets infrastructure consumers use the same n-by-n request, buffered-response, and streaming behavior instead of maintaining downstream translation forks.

This is the Switchyard-side companion to NVIDIA/NeMo-Relay #369. Relay consumes this crate only when its native Switchyard plugin feature is enabled; routing-service lifecycle, credentials, dispatch, retries, and observability remain Relay-owned.

The PR is based on topic/nemo-relay-integration after #35, keeping StageRouter baseline/freshness policy separate from translation behavior.

How tested

  • cargo fmt --all -- --check
  • cargo test -p switchyard-translation — 61 passed
  • cargo clippy -p switchyard-translation --all-targets -- -D warnings
  • cargo test -p switchyard-translation in rust:1.93.0 — 61 passed
  • Downstream Relay builds both without and with its optional switchyard feature
  • cargo test -p nemo-relay-switchyard — 31 passed
  • Relay Switchyard process E2E — buffered, streaming, and fail-open paths passed

Checklist

  • Unit tests added for corrected translation behavior.
  • Existing public translation entry points remain intact.
  • No routing, server, credential, or dispatch behavior is introduced.
  • Commits signed off per the DCO.
  • Python-only class/export checklist items are not applicable to this Rust crate.
  • No customer-facing CLI or Python surface changed.

Notes for reviewers

Suggested review order:

  1. Request image and tool-argument normalization in the buffered codecs.
  2. Multi-output buffered response encoding for Chat and Anthropic targets.
  3. Stream failure and incomplete-terminal behavior.
  4. The translation-matrix regression fixtures.
  5. The crate-scoped Rust 1.93 compatibility override.

The change deliberately remains protocol-neutral. Relay-specific target binding, fallback policy, and unsupported-extension fail-open rules stay outside this crate.

@bbednarski9
feat: harden cross-protocol translation
88e95b2 
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@bbednarski9
bbednarski9 marked this pull request as ready for review July 9, 2026 20:45
@bbednarski9
bbednarski9 requested a review from a team as a code owner July 9, 2026 20:45
@bbednarski9
fix: preserve incomplete stream terminals
d8f266a 
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@bbednarski9 bbednarski9 changed the title feat: harden cross-protocol translation feat(translation): harden cross-protocol translation Jul 9, 2026
@bbednarski9
bbednarski9 marked this pull request as draft July 9, 2026 20:48
@bbednarski9
bbednarski9 marked this pull request as ready for review July 9, 2026 20:52
@bbednarski9
bbednarski9 merged commit 8f9db9a into NVIDIA-NeMo:topic/nemo-relay-integration Jul 9, 2026
16 checks passed
This was referenced Jul 9, 2026
Merged
Merged
rapids-bot Bot pushed a commit to NVIDIA/NeMo-Relay that referenced this pull request Jul 13, 2026
@bbednarski9
feat(plugin)!: Add Switchyard Decision API routing plugin (#369)
658f287 
> [!WARNING]
> **BREAKING CHANGE:** **[Rust source compatibility]** This PR adds `FlowError::Upstream` and new fields to the public `AtofEndpointConfig`, `AtofEndpointSectionConfig`, and `EmitMarkEventParams` types. Downstream exhaustive `FlowError` matches must handle the new variant, and direct struct literals must initialize the new fields or migrate to the provided constructor/builder APIs. Builder-based mark emission and default Relay configurations remain compatible.

#### Overview

This adds Relay's first-party integration with the [Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API. The plugin routes buffered and streaming LLM requests through an external Switchyard service, executes the selected Relay-owned backend, and records causal model-routing optimization evidence.

Switchyard owns provider-protocol translation through its `switchyard-translation` Rust library. Relay consumes that library only when the native Switchyard plugin is enabled; default Relay builds do not include either the plugin or the translation dependency.

- [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license.
- [x] I searched existing issues and open pull requests, and this does not duplicate existing work.

#### Details

##### Plugin and execution behavior

- Add the standalone `nemo-relay-switchyard` first-party plugin.
- Normalize stable Relay identity and build `switchyard.routing_request.v1` requests.
- Validate selected routes and optional counterfactual baselines against exact Relay-owned backend bindings.
- Support enforce and observe-only modes, six request-materialization modes, bounded retry-aware dispatch, protocol-specific trusted fallbacks, and no retry after the first upstream stream item.
- Record one model-routing optimization contribution only after the selected route commits. Failed decisions, failed provider attempts, and fallback dispatches do not claim routing savings.
- Preserve Switchyard routing metadata, including snapshot freshness, decision source, event count, and turn depth.
- Continue using Relay's existing ATOF exporter and lifecycle; the plugin does not install local routers, feature accumulators, or a duplicate event subscriber.

##### Translation ownership and optional dependency

Request, buffered-response, and SSE translation across OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages use Switchyard's public `switchyard-translation` library instead of a Relay-local translation engine.

The dependency is pinned to the immutable topic-branch commit [`8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8`](NVIDIA-NeMo/Switchyard@8f9db9a), which includes the Decision API series, StageRouter baseline/freshness behavior, and [Switchyard #46](NVIDIA-NeMo/Switchyard#46).

The Relay CLI exposes this through `--features switchyard`:

- Without that feature, `nemo-relay-cli` contains neither `nemo-relay-switchyard` nor `switchyard-translation`.
- With that feature, the plugin and its pinned Switchyard translation library are compiled and registered.
- Relay continues to own target bindings, credentials, dispatch, retries, fallbacks, lifecycle, and observability.
- Relay retains narrow fail-open policy checks for provider extensions that cannot be translated without data loss.

##### Runtime requirements

The current experimental deployment requires a separately running Switchyard service from the pinned topic commit. The plugin does not start or supervise that service.

Switchyard availability is not a Relay startup requirement. It is a request-time dependency: connection failures, timeouts, malformed decisions, selected-route drift, unsupported extensions, and missing required identity fail open to the configured same-protocol default. ATOF-backed profiles additionally require Relay's ATOF HTTP exporter to target Switchyard's `/v1/atof/events` endpoint with preserved field names and environment-referenced authentication.

The public examples document this service boundary, verify the exact Switchyard revision before startup, and intentionally remain manual compatibility workflows rather than production orchestration.

##### Optimization accounting and observability

This PR builds on the plugin-neutral accounting contract merged in [NeMo Relay #389](#389). Applied routing contributions carry the selected and baseline model transition, decision identity, backend and tier, attempt, rollout mode, reason, and router metadata. Observe-only decisions remain visible as non-applied evidence.

Relay core combines routing and other plugin contributions at LLM close while retaining explicit token evidence for downstream repricing. Existing requested, decision, retry, fallback, and error routing marks remain available alongside the canonical optimization contribution.

The generic ATIF change that attributes `step.model_name` to the effective response model is intentionally separated into [NeMo Relay #399](#399). This PR still ensures translated provider responses carry the actual selected model; #399 independently controls how ATIF presents it.

##### Testing summary

- `cargo fmt --all -- --check`
- `cargo clippy -p nemo-relay-switchyard --all-targets -- -D warnings`
- `cargo clippy -p nemo-relay-cli --features switchyard --all-targets -- -D warnings`
- `cargo check -p nemo-relay-cli` — proves the default CLI builds without Switchyard.
- `cargo check -p nemo-relay-cli --features switchyard` — proves the opt-in dependency path.
- `cargo test -p nemo-relay-switchyard` — 31 passed.
- `cargo test -p nemo-relay-cli gateway::tests` — 33 passed without the feature.
- `cargo test -p nemo-relay-cli --features switchyard gateway::tests` — 33 passed with the feature.
- `cargo test -p nemo-relay-cli --features switchyard --test switchyard_process_e2e` — buffered, streaming, retry, and fail-open process boundary passed after the split.
- `examples/switchyard/run-real-e2e.sh` against the exact pinned Switchyard service — passed with the expected StageRouter route sequence `provider/weak`, `provider/strong`, `provider/strong`.
- The pinned Switchyard translation crate separately passes 61 request, response, streaming, extension, and round-trip tests under Rust 1.93.

##### Previous feedback addressed

- Split the implementation at repository ownership boundaries.
- Replaced Relay's custom translation engine with the Switchyard-owned reusable library.
- Kept the dependency compile-time optional for users who do not enable the Relay-native Switchyard plugin.
- Removed in-process Random, LLM, and Cascade routers, Relay-side feature accumulation, duplicate Switchyard subscriptions, and the global fallback target.
- Replaced literal credentials with environment-referenced headers and activation-time validation.
- Hardened internal dispatch headers and structured upstream-failure metadata.
- Standardized routing terminology on StageRouter.
- Standardized model-routing evidence on Relay's shared optimization contract instead of adding a second post-call savings mark.
- Moved generic ATIF effective-model attribution into focused PR #399.
- Moved Fern documentation into focused experimental docs PR #400 so this functional PR remains code-focused.
- Kept public examples free of internal-only providers, credentials, Visor source references, and private infrastructure assumptions.

#### Where should the reviewer start?

1. `crates/switchyard/src/component.rs` — Decision API lifecycle, exact target and baseline validation, retries, fallback, and terminal route commitment.
2. `crates/switchyard/src/translation.rs` and `stream_translation.rs` — thin policy adapters over the Switchyard-owned translation library.
3. `crates/cli/Cargo.toml` and the `cfg(feature = "switchyard")` registration paths — compile-time optionality.
4. `crates/cli/src/gateway.rs` — reserved dispatch headers and structured retry-aware upstream failures.
5. `crates/cli/tests/switchyard_process_e2e.rs` — process-boundary buffered, streaming, and fail-open coverage.
6. `examples/switchyard/README.md` and `run-real-e2e.sh` — pinned experimental service workflow.

#### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

- Relates to [NVIDIA/NeMo-Relay #389](#389).
- Relates to [NVIDIA/NeMo-Relay #399](#399).
- Relates to [NVIDIA/NeMo-Relay #400](#400).
- Relates to [NVIDIA-NeMo/Switchyard #46](NVIDIA-NeMo/Switchyard#46).
- Paired with the Switchyard Decision API topic series:
  - [NVIDIA-NeMo/Switchyard #8](NVIDIA-NeMo/Switchyard#8)
  - [NVIDIA-NeMo/Switchyard #9](NVIDIA-NeMo/Switchyard#9)
  - [NVIDIA-NeMo/Switchyard #11](NVIDIA-NeMo/Switchyard#11)
  - [NVIDIA-NeMo/Switchyard #35](NVIDIA-NeMo/Switchyard#35)

Authors:
  - Bryan Bednarski (https://github.com/bbednarski9)

Approvers:
  - Will Killian (https://github.com/willkill07)
  - Maryam Najafian (https://github.com/mnajafian-nv)
  - https://github.com/Salonijain27

URL: #369
@bbednarski9 bbednarski9 mentioned this pull request Jul 14, 2026
Merged
2 tasks
rapids-bot Bot pushed a commit to NVIDIA/NeMo-Relay that referenced this pull request Jul 15, 2026
@bbednarski9
feat(plugin): validate named Switchyard ATOF endpoints (#420)
04372e1 
#### Overview

Validate the ATOF delivery dependency for history-backed Switchyard profiles at Relay startup. This source change is split from #400 so the implementation and user-facing documentation can be reviewed independently.

- [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license.
- [x] I searched existing issues and open pull requests, and this does not duplicate existing work.

#### Details

- Add an optional, unique `name` to observability ATOF stream sinks and expose it through the Python, Go, and Node plugin configuration helpers.
- Replace the experimental Switchyard component's duplicated `atof_endpoint_url` setting with `atof_endpoint_name`.
- When `context_mode = "atof_required"`, fail startup validation unless the name resolves to exactly one enabled `type = "stream"` ATOF sink that uses `http_post`, preserves canonical field names, and uses environment-referenced authentication headers.
- Derive omitted sink transport and field-name policy values from core's public enum defaults while keeping `http_post` and `preserve` as explicit Switchyard requirements.
- Keep `payload_only` profiles independent of ATOF delivery.
- Add Rust, Python, Go, and Node coverage for stream-sink name serialization, uniqueness, context-sensitive validation, and disabled-protocol defaults.
- Update all three Switchyard example plugin configurations to use observability config v2 `sinks`, including a named stream sink.
- Keep the corresponding user-facing guides and README changes in #400.

**Breaking change:** Experimental Switchyard configurations must replace `atof_endpoint_url` with `atof_endpoint_name`, and the corresponding observability ATOF stream sink must define the same `name`. Rust struct literals for `AtofStreamSinkSectionConfig` must initialize the new optional `name` field.

#### Where should the reviewer start?

1. `crates/switchyard/src/component.rs` — context-sensitive cross-component startup validation.
2. `crates/switchyard/tests/unit/component_tests.rs` — accepted and rejected ATOF configurations.
3. `crates/core/src/observability/plugin_component.rs` — named stream-sink configuration and uniqueness validation.
4. `python/nemo_relay/observability.py`, `go/nemo_relay/observability_plugin.go`, and `crates/node/observability.d.ts` — typed plugin helpers for named stream sinks.
5. `examples/switchyard/plugins.toml` — minimal named-stream-sink configuration.

#### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

- Relates to #400.
- Relates to #369.
- Relates to NVIDIA-NeMo/Switchyard#46.

#### Validation

- `cargo test --workspace --quiet` — full Rust workspace passed.
- `uv run pytest -q python/tests --ignore=python/tests/integrations` — 528 passed.
- `npm test --workspace=nemo-relay-node` — 264 passed.
- `cargo build --release -p nemo-relay-ffi` followed by `go test -v ./...` with the FFI library path configured — all Go packages passed.
- `cargo clippy --workspace --all-targets -- -D warnings` — passed.
- `uv run pre-commit run --all-files` with the `just`-dependent protobuf hook and unchanged-lockfile attribution generators skipped — all remaining hooks passed.
- `git diff --check` — passed.




## Summary by CodeRabbit

- **New Features**
  - Added optional `name` identifiers for ATOF stream endpoint/sink configuration across exported types and schemas.
  - Switchyard now selects the ATOF stream endpoint by `atof_endpoint_name` (instead of URL-based selection).

- **Bug Fixes**
  - Added validation for endpoint `name` (must be non-empty, trim-safe, and unique) and improved diagnostics for invalid values.
  - Strengthened “ATOF required” checks to ensure the selected named endpoint matches expected transport/policy/header requirements.

- **Documentation / Examples**
  - Updated example configuration files to use `atof_endpoint_name` and per-sink endpoint `name`.

Authors:
  - Bryan Bednarski (https://github.com/bbednarski9)

Approvers:
  - Maryam Najafian (https://github.com/mnajafian-nv)
  - Will Killian (https://github.com/willkill07)

URL: #420
@ryan-lempka ryan-lempka mentioned this pull request Jul 15, 2026
Draft
rapids-bot Bot pushed a commit to NVIDIA/NeMo-Relay that referenced this pull request Jul 15, 2026
@bbednarski9
docs: document experimental Switchyard integration (#400)
e512afe 
#### Overview

Document the experimental Switchyard integration, including its service boundary, Relay-owned dispatch model, target validation, fail-open behavior, and ATOF delivery requirements.

The source implementation and tests landed in #420. This documentation PR is now based on `main` after the Switchyard publication, activation health-check, and same-protocol routing follow-ups.

> **Experimental:** The integration is early access, disabled in default Relay builds, and currently requires a separately running Switchyard service.

- [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license.
- [x] I searched existing issues and open pull requests, and this does not duplicate existing work.

#### Details

- Add `Configure Plugins → Switchyard (Experimental)` to the Fern navigation.
- Document the complete component configuration, deployment boundary, retries, target validation, streaming behavior, and experimental limitations.
- Clarify that the Switchyard service selects a backend while Relay validates the decision, translates requests and responses in process, and dispatches directly to Relay-owned targets.
- Document the distinction between a local ATOF file sink and the named HTTP stream sink that feeds Switchyard's accumulator.
- Document that StageRouter configurations declare `context_mode = "atof_required"` and configure the matching named observability stream sink.
- Correct the pinned Switchyard checkout instructions and document defaults only for enabled inbound protocols.
- Document `header_env` for ATOF stream sinks and recommend environment-referenced authentication.
- Keep all implementation and test changes in #420.
- Keep the runnable Switchyard TOML configurations with the source change in #420.

#### Where should the reviewer start?

1. `docs/configure-plugins/switchyard/configuration.mdx` — complete configuration and runtime behavior.
2. `docs/configure-plugins/switchyard/about.mdx` — user-facing architecture and prerequisites.
3. `examples/switchyard/README.md` — pinned compatibility workflow.

#### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

- Relates to #420.
- Relates to #369.
- Relates to NVIDIA-NeMo/Switchyard#46.

#### Validation

- Documentation reference generation completed successfully.
- `fern check --warnings` — zero errors; the unauthenticated redirect check was skipped with one warning.
- `fern docs broken-links --strict` — passed.
- All seven changed TOML examples parse successfully.
- The Python and Node.js ATOF configuration examples validate with zero diagnostics.
- Targeted pre-commit checks passed for all changed documentation and README files.
- `git diff --check origin/main -- <changed-files>`




## Summary by CodeRabbit

* **Documentation**
  * Added documentation for the experimental Switchyard integration, including setup, configuration, routing behavior, fallbacks, and compatibility requirements.
  * Added Switchyard to the plugin configuration navigation and built-in plugin list.
  * Expanded ATOF endpoint documentation with naming and environment-based header configuration.
  * Updated Switchyard examples and setup guidance, including ATOF-backed routing requirements and runtime behavior.

Authors:
  - Bryan Bednarski (https://github.com/bbednarski9)

Approvers:
  - https://github.com/lvojtku
  - Will Killian (https://github.com/willkill07)

URL: #400
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bbednarski9