feat(realtime): speaker-aware conversations - surface identity to client and LLM

[localai-bot]

Speaker-aware realtime conversations

Extends the realtime voice-recognition gate so the recognized speaker is surfaced to the client and fed to the LLM for personalized replies (e.g. "Hey Jeremy, ..."), decoupled from the authorization gate and fully backward compatible.

What it does

• Stream identity to the client via a new server event conversation.item.speaker (a LocalAI extension; OpenAI clients ignore unknown event types, so it is non-breaking):

  { "type": "conversation.item.speaker", "item_id": "...",
    "speaker": { "name": "Jeremy", "id": "spk_1", "confidence": 92.0, "distance": 0.1, "matched": true } }

• Inform the LLM (hybrid, configurable): sets the per-message OpenAI name field on each user turn and/or appends a The current speaker is <Name>. note to the system message. Each user item carries its own speaker, so multi-speaker histories stay correctly attributed.
• Decoupled, configurable new keys under pipeline.voice_recognition:
  • enforce (default true): the authorization gate. false resolves/surfaces the speaker without ever dropping a turn.
  • identity.{announce, announce_unknown, personalize, inject_name, inject_system_note, note_unknown}: surfacing + personalization, independent of enforce. When set, identity is resolved on every turn even under when: first.

Design

• voiceGate is split into Resolve (embed once → types.Speaker identity) and a pure authorize policy step; the legacy Authorize wrapper is preserved so existing behavior and tests are unchanged.
• One embed per turn feeds the gate, the client event, and personalization.
• The gate still fails closed: when enforcing, a denied or resolve-errored utterance never reaches the LLM/tools/TTS. enforce: false never drops a turn.

Backward compatibility

A voice_recognition config with neither enforce nor identity behaves exactly as before. The new event and item field are additive.

Tests

TDD throughout (Ginkgo/Gomega). New unit + integration specs cover: the Resolve/authorize split and confidence math, event marshaling, per-turn surfacing, enforce:false never dropping, when:first×identity per-turn re-resolution, and multi-speaker history attribution. core/config and core/http/endpoints/openai both green.

Docs

docs/content/features/openai-realtime.md extended with the new config keys and the conversation.item.speaker event.

Assisted-by: Claude:claude-opus-4-8

🤖 Generated with Claude Code