Skip to content

feat(libsy-server): demo libsy + random routing + affinity + builtin proxy#86

Draft
messiaen wants to merge 15 commits into
grclark/llm-client-cratefrom
grclark/libsy-proxy
Draft

feat(libsy-server): demo libsy + random routing + affinity + builtin proxy#86
messiaen wants to merge 15 commits into
grclark/llm-client-cratefrom
grclark/libsy-proxy

Conversation

@messiaen

Copy link
Copy Markdown
Contributor

Summary

Introduces libsy-server, a minimal axum proxy that fronts coding agents
(Claude Code, Codex) and A/B-routes each request between a weak and strong
upstream tier using libsy's RandomAlgo. Also merges in agent-aware routing /
affinity
(from agent/subagent-aware-routing) and fixes two Anthropic→OpenAI-
Responses translation bugs that broke tool-using agent turns.

What's included

New crate — libsy-server (crates/libsy-server/)

  • HTTP surface for OpenAI Chat, OpenAI Responses, and Anthropic Messages
    (/v1/chat/completions, /v1/messages, /v1/responses, /v1/models, /health).
  • Decode → route via RandomAlgo → translate to the chosen tier's format →
    translate back to the inbound format. Buffered and streaming (SSE) supported.
  • CLI: --weak/--strong + --weak-format/--strong-format (per-tier upstream wire
    format, credential picked by provider), --affinity [session|subagent],
    --log-routing, --host/--port.
  • Populates request metadata from headers so affinity can key on it; recognizes
    Claude Code's X-Claude-Code-Session-Id.
  • Caller auth headers are dropped and the real backend credential injected
    (sentinel-key friendly).
  • README with quickstart + Claude Code setup; integration tests via
    tower::oneshot + wiremock.

Agent-aware routing / affinity (crates/libsy/, merged)

  • affinity module: Affinity trait, SessionAffinity, SubAgentAffinity, and
    metadata_from_headers (Switchyard/Codex/Relay/Dynamo/Claude Code header
    normalization).
  • RandomAlgo::with_affinity(...); AgentContext on protocol Metadata.
  • demo/libsy-proxy POC.

Translation fixes (crates/switchyard-translation/)

  • Anthropic tool_use → Responses function_call now emits arguments as a
    JSON string (was an object → API 400).
  • Anthropic thinking → Responses now dropped instead of encoded as a
    content-bearing reasoning input item (was → API 400), matching the OpenAI Chat
    encoder. Reasoning-only messages emit nothing; private reasoning never leaks into
    content.

Testing

  • libsy-server: 9 integration tests (buffered/streaming per format, cross-format
    routing, error mapping).
  • switchyard-translation: existing suite green + 2 new regression tests for the
    arguments/reasoning fixes.
  • libsy: affinity unit/integration tests (incl. Claude Code session header).
  • Verified end-to-end with Claude Code: streaming, tool use, and multi-turn work
    across both tiers; concurrent sessions each pin to one tier under
    --affinity session (deterministic: A→weak 12/12, B→strong 12/12).

Notes for reviewers

  • Behavior change: with per-tier formats the proxy accepts any inbound endpoint
    and translates to the tier's format (no longer same-format-only), so there's no
    "unsupported inbound format" 400.
  • Affinity is all-or-nothing per session — a session pinned to the weak tier
    runs the whole task on it (intended for per-session A/B measurement).
  • One change touches the just-merged metadata_from_headers to recognize
    X-Claude-Code-Session-Id.

Checklist

  • One class per file; filename = snake_case of the primary class.
  • New public symbols exported from switchyard/__init__.py.__all__ if intended for downstream use.
  • Unit tests added for new components / bug fixes.
  • README / --help updated if customer-facing surface changed.
  • Commits signed off (Signed-off-by: Your Name <email>) per the DCO.

Notes for reviewers

Anything reviewers should pay extra attention to — risky paths, follow-up tickets, intentional trade-offs.

messiaen and others added 14 commits July 16, 2026 19:39
Signed-off-by: Greg Clark <grclark@nvidia.com>
…eutral IR

Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
@messiaen messiaen changed the title Grclark/libsy proxy feat(libsy-server): demo libsy + random routing + affinity + builtin proxy Jul 17, 2026
Signed-off-by: ayushag <ayushag@nvidia.com>
@messiaen
messiaen force-pushed the grclark/llm-client-crate branch from 30d8979 to a3c4498 Compare July 17, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants