Skip to content

feat(openai): route OpenAI Realtime (WebSocket) through LLM Gateway#94

Open
cosminacho wants to merge 1 commit into
mainfrom
feat/openai-realtime
Open

feat(openai): route OpenAI Realtime (WebSocket) through LLM Gateway#94
cosminacho wants to merge 1 commit into
mainfrom
feat/openai-realtime

Conversation

@cosminacho

Copy link
Copy Markdown
Collaborator

What & why

Adds OpenAI Realtime API (WebSocket) support, routed through the UiPath LLM Gateway. UiPathOpenAI / UiPathAsyncOpenAI now expose client.realtime.connect() exactly like the stock OpenAI SDK:

client = UiPathAsyncOpenAI(model_name="gpt-realtime")
async with client.realtime.connect() as conn:
    await conn.session.update(session={"type": "realtime", "output_modalities": ["text"]})
    await conn.conversation.item.create(item={
        "type": "message", "role": "user",
        "content": [{"type": "input_text", "text": "Say hello!"}],
    })
    await conn.response.create()
    async for event in conn:
        if event.type == "response.output_text.delta":
            print(event.delta, end="")
        elif event.type == "response.done":
            break

How it works

The Realtime API speaks over a WebSocket, so it can't use the existing httpx routing. A .realtime cached-property override returns a gateway-aware resource wrapper that, on connect():

  • points the client's websocket_base_url at the gateway passthrough realtime path (.../vendor/nativeopenai/model/<model>/realtime),
  • sets the S2S bearer token as api_key (read straight from LLMGatewayS2SAuth.access_token; the SDK sends it as Authorization: Bearer on the WebSocket upgrade),
  • injects the X-UiPath-* routing headers on the upgrade.

Completions/embeddings are unaffected — their auth comes from the httpx auth pipeline, and the realtime URL is built lazily only when .realtime is accessed (so platform/agenthub construction is untouched).

Why no separate class / no LangChain wrapper

The openai-python README shows realtime works on the standard client — no subclass needed — so it's folded into the existing clients. LangChain has no realtime chat-model abstraction (langchain#28086 is still open; ChatOpenAI has no .realtime), so there's nothing to wrap there — realtime is used from a LangChain project by dropping down to the core client.

Packages affected

Both (version together):

  • core1.15.0: realtime clients in uipath.llm_client.clients.openai; openai extra now installs openai[realtime] (pulls in websockets); helper build_realtime_ws_base_url.
  • langchain1.15.0: openai extra now also pulls uipath-llm-client[openai] so realtime works out of the box from a uipath-langchain-client[openai] install. No code change.

Tests / checks

  • New unit tests in tests/core/clients/openai/test_realtime_unit.py (URL helper, lazy build, websocket_base_url + token + header wiring, default-model resolution, token refresh). Realtime is a WebSocket so it can't be VCR-recorded; end-to-end was verified manually against alpha (gpt-realtime).
  • ruff check, ruff format --check, pyright all clean; pytest tests → 1968 passed, 0 failures.

🤖 Generated with Claude Code

UiPathOpenAI / UiPathAsyncOpenAI now expose `client.realtime.connect()`
exactly like the stock OpenAI SDK, opening a WebSocket to the gateway's
passthrough realtime endpoint (.../vendor/<vendor>/model/<model>/realtime).
The .realtime resource points websocket_base_url at the gateway, sets the
S2S bearer token as api_key (sent as Authorization: Bearer on the upgrade),
and injects the X-UiPath-* routing headers. Completions/embeddings are
unaffected: their auth uses the httpx pipeline, and the realtime URL is
built lazily on .realtime access.

- openai extra now installs openai[realtime] (pulls in websockets)
- langchain openai extra pulls core[openai] so realtime works from a
  langchain install (LangChain has no realtime chat-model abstraction)
- bump core + langchain to 1.15.0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant