Ensure input stream is only tee'd when it's actually being used - #1088
Conversation
🦋 Changeset detectedLatest commit: b1c8ba6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
toubatbrian
left a comment
toubatbrian
left a comment
There was a problem hiding this comment.
LGTM. qq: is this change addressing a specific bug, or is it mainly a general improvement?
see the linked issue #1070 |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (this.realtimeSession && this.audioRecognition) { | ||
| const [realtimeAudioStream, recognitionAudioStream] = this.audioStream.stream.tee(); | ||
| this.realtimeSession.setInputAudioStream(realtimeAudioStream); | ||
| } | ||
|
|
||
| if (this.audioRecognition) { | ||
| this.audioRecognition.setInputAudioStream(recognitionAudioStream); | ||
| } else if (this.realtimeSession) { | ||
| this.realtimeSession.setInputAudioStream(this.audioStream.stream); | ||
| } else if (this.audioRecognition) { | ||
| this.audioRecognition.setInputAudioStream(this.audioStream.stream); |
There was a problem hiding this comment.
🔴 aecWarmupAudioFilter is defined but never applied to the audio stream
The refactored code creates the aecWarmupAudioFilter TransformStream (lines 492-500) but never pipes the audio stream through it. The old code applied the filter via .pipeThrough(aecWarmupAudioFilter) before .tee(), which discarded audio frames during AEC warmup (when the agent is speaking and _aecWarmupRemaining > 0). In the new code, all three branches pass this.audioStream.stream (or its .tee()) directly to consumers without the filter. This means audio frames that should be discarded during AEC warmup will now be forwarded to realtimeSession.setInputAudioStream() and audioRecognition.setInputAudioStream(), potentially causing echo/feedback issues.
Prompt for agents
In agents/src/voice/agent_activity.ts, the attachAudioInput method (around line 484) defines an aecWarmupAudioFilter TransformStream but never uses it. The audio stream must be piped through this filter before being consumed. All three branches (lines 504-512) need to apply the filter:
1. For the tee branch (realtimeSession && audioRecognition): pipe through filter first, then tee:
const [realtimeAudioStream, recognitionAudioStream] = this.audioStream.stream.pipeThrough(aecWarmupAudioFilter).tee();
2. For realtimeSession-only branch: pipe through filter:
this.realtimeSession.setInputAudioStream(this.audioStream.stream.pipeThrough(aecWarmupAudioFilter));
3. For audioRecognition-only branch: pipe through filter:
this.audioRecognition.setInputAudioStream(this.audioStream.stream.pipeThrough(aecWarmupAudioFilter));
Note: The comment on lines 488-491 explains why the filter is applied on the downstream stream rather than on the source audioStream. Make sure the filter is still applied on the downstream side.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Description
closes #1070
Changes Made
Pre-Review Checklist
Testing
restaurant_agent.tsandrealtime_agent.tswork properly (for major changes)Additional Notes
Note to reviewers: Please ensure the pre-review checklist is completed before starting your review.