feat(voice): fire _initialize_conversation as background task on WebSocket open - #100
Open
bsahajsinghani wants to merge 18 commits into
Open
feat(voice): fire _initialize_conversation as background task on WebSocket open#100bsahajsinghani wants to merge 18 commits into
bsahajsinghani wants to merge 18 commits into
Conversation
- Fix create_observation() payload format: wrap in {"observations": [...]}
so Memora API accepts the request (was sending flat dict → silent 400)
- Inject Memora memory into agent instructions via MemoryPromptBuilder.compose()
so the LLM actually uses observations/summaries from past calls
- Enable memory_mode="always" in voice_streaming.py example
- Wire up CI webhook: if CONVERSATION_INTELLIGENCE_CONFIGURATION_ID is set,
expose /ci-webhook route for near real-time custom operator ingestion
- Auto-close conversation via CO API on WebSocket disconnect so Memora
extraction triggers immediately instead of waiting for inactive timeout
- Add CI/memory section to getting_started/README.md explaining standard
vs alternate write paths, latency tips, and links to lifecycle docs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
**STT model selection:**
- Add `transcription_provider` and `speech_model` fields to VoiceChannelConfig,
TwiMLOptions, and VoiceChannel so callers can pass e.g.
`{"transcription_provider": "deepgram", "speech_model": "flux"}` at construction
- Wire both fields through twiml.py → ConversationRelay attributes
(`transcriptionProvider`, `speechModel`)
- Deepgram Flux is turn-aware: waits for a complete utterance before emitting
a transcript, eliminating spurious mid-utterance STT chunks and the redundant
Recall+LLM calls they caused (~6:1 chunk:turn → 1:1 with Flux)
- Document all supported Deepgram and Google models in getting_started/README.md
with a note about the error-64101 pitfall for invalid model names
**OTel tracing:**
- Add src/tac/tracing.py: lightweight OTel setup module controlled by
OTEL_ENABLED / OTEL_ENDPOINT / OTEL_SERVICE_NAME env vars
- Provides start_call(), end_call(), turn_span(), memory_span(), llm_span(),
inject_traceparent() context managers used in channel.py
- Wire traceparent header propagation into BaseAPIClient so Recall HTTP calls
carry the active trace context (enables Memora correlation on Grafana)
- Export `tracing` from tac.__init__ so examples can call tracing.setup_tracing()
- Add LATENCY_OBSERVABILITY_PLAN.md: phased plan for full end-to-end latency
measurement (Jaeger local → missing spans → Voice Insights → VAD TTFA)
**STT chunk diagnostics (debug-only):**
- [STT] / [STT TURN N] / [STT SUMMARY] print statements in channel.py to
confirm chunk:turn ratio during benchmarking; not intended for production
Note: This is a personal benchmarking branch — no review needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fills in previously untracked timing gaps in the voice call lifecycle: - call.co_init: measures time TAC spends waiting for Conversation Orchestrator to create the conversation after ConversationRelay connects. This polling phase can take several hundred milliseconds depending on CO load and was previously invisible. - call.profile_lookup: measures the participant fetch and customer address resolution that happens immediately after conversation init. Covers the list_participants() HTTP call and customer profile ID resolution. - record_first_token(): adds a first_token_sent point-in-time event to the llm.completion span the moment the first streaming token is written to the WebSocket. This splits the LLM span into two readable phases: time waiting for the first token (TTFT) and time streaming the full response. Both co_init and profile_lookup are parented to the call root span rather than the turn span, since they only occur once at call start. Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Records two actionable findings from Phase 1-2 Jaeger traces: - CO init + profile lookup accounts for ~1s at call start - memory.recall TAC-side is 4-7x Memora server-side; switching memory_mode to "once" would save ~700ms/turn from turn 2 onwards Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Notes that the memory.recall overhead is not a region mismatch — TAC and Memora are co-located in us-east-1 (intra-AZ). The ~280ms gap per recall is TLS connection setup paid fresh on every call. Connection pooling + memory_mode "once" together would reduce per-turn recall from ~700ms to near zero after turn 1. Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rgets Added industry-standard latency thresholds, full experiment configuration table aligned with the intern project proposal, and additional optimization ideas to explore beyond the core experiments. Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… coverage Breaks down the two previously opaque spans into individually measurable steps: memory.recall now contains: - memory.profile_lookup — lookup_profile() HTTP call - memory.profile_fetch — get_profile() HTTP call to fetch traits - memory.recall_api — the actual Memora vector search call llm.completion now contains: - llm.prompt_build — MemoryPromptBuilder + message deepcopy - llm.response_stream — WebSocket streaming loop, first to last token sent Note: personal benchmarking branch — no review needed.
- Move OTel packages to optional tac[tracing] extra — no longer a hard
dependency for all TAC users
- Auto-initialize tracing in TAC.__init__ when OTEL_ENABLED=true — zero
manual setup required
- Change default OTEL_ENDPOINT from Twilio-internal URL to
http://localhost:4318 (standard OTel collector port, works with any
backend — Jaeger, Grafana Tempo, Honeycomb, Datadog)
- Graceful ImportError handling in setup_tracing() with actionable warning
when packages not installed
- Add test_tracing.py: 19 tests covering no-op behaviour, start/end call
lifecycle, all span types, turn counter, and traceparent injection
Span hierarchy for a voice call:
call → call.co_init, call.profile_lookup, call.first_prompt_wait
→ turn → memory.recall → memory.recall_api, memory.profile_lookup
→ llm.completion → llm.prompt_build, llm.response_stream
(event: first_token_sent)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix isort order in tac.py and tracing.py - Replace typing.Iterator with collections.abc.Iterator (UP035) - Remove unused NonRecordingSpan import - Remove unused os/MagicMock imports in test_tracing.py - Update module docstring to reflect new default endpoint Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wraps tac.list_observations() with an OTel child span so latency of the observations fetch is visible in Jaeger alongside co_init and profile_lookup. Adds record_obs_fetch_count() to emit an observations.fetched event with the actual count loaded. Adds two test coverage cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use call_sid from session metadata in _cleanup_connection so end_call always uses the same key that start_call was registered with (fixes orchestrated mode where conv_id != call_sid) - Close span directly in finally block when conv_id is None (user hangs up before first prompt) to prevent span leak
…ocket open Moves CO polling + profile lookup out of the first-prompt critical path. The task starts immediately after setup_msg so init runs during user speech; first_prompt_wait_span measures actual blocking time at the await site. Adds cancellation and failure handling in the finally block, plus two tests covering early hangup and task failure scenarios.
bsahajsinghani
requested review from
ryanrishi,
ryanrouleau and
xinghaohuang91
as code owners
August 7, 2026 06:38
…oryClient Add warning log to _initialize_conversation when CO polling exceeds 800ms threshold — makes slow CO startups visible in structured logs without needing to open Jaeger. Add list_observations method to MemoryClient (GET Observations endpoint) with created_before param to freeze the observation snapshot at call-start time. Thread created_before through TAC.list_observations so callers can prevent mid-call CO updates from changing the memory block injected into later turns. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reverts memory.py and tac.py to pre-change state. The created_before param on list_observations was incomplete — base.py would also need updating for it to have any effect. Pulling that work until it can be done properly end-to-end. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously call_sid was only written to session.metadata inside the prompt handler, so if a caller hung up before speaking the cleanup path called tracing.end_call(conv_id) instead of end_call(call_sid), leaving the root span open. Moving the write into _initialize_conversation ensures it is always set before any disconnect can occur. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves CO polling + profile lookup out of the first-prompt critical path by firing
_initialize_conversationas anasyncio.create_taskimmediately aftersetup_msg. On first prompt, the handlerawaits the task — if it finished during user speech, wait time is 0ms.Depends on PR #99 (OTel tracing layer) — uses
tracing.first_prompt_wait_span.Experiment results (Deepgram Flux, gpt-4o-mini,
memory_mode=never, 4-turn scenario, 6 runs excluding warmup):The
first_prompt_wait=0mson all runs confirms init consistently finishes during user speech even on cold CO (~3.5s startup).Scope: turn-1 latency only. Per-turn latency (turns 2–N) is dominated by LLM TTFT and memory strategy, not init overhead.
Risks handled:
init_task.cancel()+awaitinfinallyblockconv_idnever set: cleaned up in finally blockcall_sidisstr | NoneonSetupMessage: guarded withif call_sid and ...beforecreate_taskType of Change
Checklist
SDK Parity