feat(libsy-server): demo libsy + random routing + affinity + builtin proxy#86
Draft
messiaen wants to merge 15 commits into
Draft
feat(libsy-server): demo libsy + random routing + affinity + builtin proxy#86messiaen wants to merge 15 commits into
messiaen wants to merge 15 commits into
Conversation
Signed-off-by: Greg Clark <grclark@nvidia.com>
…eutral IR Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
messiaen
force-pushed
the
grclark/llm-client-crate
branch
from
July 17, 2026 21:02
30d8979 to
a3c4498
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces
libsy-server, a minimal axum proxy that fronts coding agents(Claude Code, Codex) and A/B-routes each request between a weak and strong
upstream tier using libsy's
RandomAlgo. Also merges in agent-aware routing /affinity (from
agent/subagent-aware-routing) and fixes two Anthropic→OpenAI-Responses translation bugs that broke tool-using agent turns.
What's included
New crate —
libsy-server(crates/libsy-server/)(
/v1/chat/completions,/v1/messages,/v1/responses,/v1/models,/health).RandomAlgo→ translate to the chosen tier's format →translate back to the inbound format. Buffered and streaming (SSE) supported.
--weak/--strong+--weak-format/--strong-format(per-tier upstream wireformat, credential picked by provider),
--affinity [session|subagent],--log-routing,--host/--port.Claude Code's
X-Claude-Code-Session-Id.(sentinel-key friendly).
tower::oneshot+wiremock.Agent-aware routing / affinity (
crates/libsy/, merged)affinitymodule:Affinitytrait,SessionAffinity,SubAgentAffinity, andmetadata_from_headers(Switchyard/Codex/Relay/Dynamo/Claude Code headernormalization).
RandomAlgo::with_affinity(...);AgentContexton protocolMetadata.demo/libsy-proxyPOC.Translation fixes (
crates/switchyard-translation/)tool_use→ Responsesfunction_callnow emitsargumentsas aJSON string (was an object → API 400).
thinking→ Responses now dropped instead of encoded as acontent-bearing
reasoninginput item (was → API 400), matching the OpenAI Chatencoder. Reasoning-only messages emit nothing; private reasoning never leaks into
content.
Testing
libsy-server: 9 integration tests (buffered/streaming per format, cross-formatrouting, error mapping).
switchyard-translation: existing suite green + 2 new regression tests for thearguments/reasoning fixes.
libsy: affinity unit/integration tests (incl. Claude Code session header).across both tiers; concurrent sessions each pin to one tier under
--affinity session(deterministic: A→weak 12/12, B→strong 12/12).Notes for reviewers
and translates to the tier's format (no longer same-format-only), so there's no
"unsupported inbound format" 400.
runs the whole task on it (intended for per-session A/B measurement).
metadata_from_headersto recognizeX-Claude-Code-Session-Id.Checklist
snake_caseof the primary class.switchyard/__init__.py.__all__if intended for downstream use.--helpupdated if customer-facing surface changed.Signed-off-by: Your Name <email>) per the DCO.Notes for reviewers
Anything reviewers should pay extra attention to — risky paths, follow-up tickets, intentional trade-offs.