Allow LemonSlice avatars to join third party meeting platforms - #6233
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| .with_identity(_MEETING_BROADCAST_IDENTITY) | ||
| .with_name(_MEETING_BROADCAST_IDENTITY) | ||
| .with_grants(grants) | ||
| .with_ttl(timedelta(hours=4)) |
There was a problem hiding this comment.
🚩 Broadcast token has 4-hour TTL without refresh mechanism
The broadcast token minted at avatar.py:189 has a fixed 4-hour TTL (timedelta(hours=4)). If a meeting runs longer than 4 hours, the token expires and the LemonSlice service would lose access to the avatar media in the LiveKit room. There's no refresh mechanism visible in the PR. For most meeting use cases this is acceptable, but it's a hard time limit worth documenting.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
The TTL is only a limit on how quickly it must be used for the avatar to join. Per LiveKit's own documentation, the avatar is not kicked from a room when the token TTL expires:
Expiration time only impacts the initial connection, and not subsequent reconnects.
longcw
left a comment
longcw
left a comment
There was a problem hiding this comment.
looks good to me! something nit:
| if frame.sample_rate == self._rate_out: | ||
| return [frame] | ||
| if self._resampler is None or self._resampler_in_rate != frame.sample_rate: | ||
| self._resampler = rtc.AudioResampler( | ||
| frame.sample_rate, self._rate_out, num_channels=frame.num_channels | ||
| ) | ||
| self._resampler_in_rate = frame.sample_rate |
There was a problem hiding this comment.
should we normalize the num_channels before sending to the STT?
There was a problem hiding this comment.
The LemonSlice relay will only ever transmit mono, but I've now added down-mixing as a defensive measure.
| backoff_time = min(backoff_time * 2, max_reconnect_delay_s) | ||
|
|
||
|
|
||
| async def stream_meeting_audio( |
There was a problem hiding this comment.
why this wrapper is needed?
There was a problem hiding this comment.
good catch. it was dead code. removed!
| async def leave_meeting(self) -> None: | ||
| """Leave the external meeting and stop the audio and chat relay.""" | ||
| meeting_bot_id = self._meeting_bot_id | ||
| session_id = self._session_id | ||
| if not meeting_bot_id or not session_id: | ||
| return | ||
|
|
||
| try: | ||
| async with LemonSliceAPI( | ||
| api_url=self._api_url, | ||
| api_key=self._api_key, | ||
| conn_options=self._conn_options, | ||
| session=self._http_session, | ||
| ) as lemonslice_api: | ||
| await lemonslice_api.leave_meeting(session_id, meeting_bot_id=meeting_bot_id) | ||
| except Exception: | ||
| logger.warning("failed to leave meeting via LemonSlice API", exc_info=True) | ||
| finally: | ||
| self._meeting_bot_id = None | ||
|
|
||
| if self._meeting_relay_stop is not None: | ||
| self._meeting_relay_stop.set() | ||
| relay_task = self._meeting_relay_task | ||
| if relay_task is not None: | ||
| relay_task.cancel() | ||
| await asyncio.gather(relay_task, return_exceptions=True) | ||
| self._meeting_relay_task = None | ||
|
|
||
| if self._meeting_chat is not None: | ||
| await self._meeting_chat.aclose() | ||
| self._meeting_chat = None | ||
|
|
||
| self._meeting_relay_stop = None |
There was a problem hiding this comment.
🚩 Meeting audio input is not restored after leave_meeting()
When join_meeting() is called, it replaces self._agent_session.input.audio with a MeetingAudioInput (avatar.py:236). However, leave_meeting() (avatar.py:267-299) never restores the original audio input. After leaving the meeting, the MeetingAudioInput remains attached but will never receive new frames (the relay task is cancelled). If the agent session continues running after leave_meeting(), the STT pipeline will hang indefinitely on await self._queue.get() (audio.py:112) since no frames are enqueued. This is fine if leave_meeting() is only called during shutdown (via aclose()), but the method is public and documented as a standalone operation, so a caller might expect the session to remain functional afterward.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Sessions are expected to be single-use in this regard. An avatar session that joins a 3rd party meeting platform should not be re-used once the avatar has left.
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| _DEFAULT_BOT_NAME = "LemonSlice Avatar" |
There was a problem hiding this comment.
🚩 MeetingChatRelay default bot name may not match actual meeting display name
When bot_name is not provided to join_meeting(), the MeetingChatRelay uses the hardcoded default "LemonSlice Avatar" (chat.py:14) for filtering self-sent chat messages. The LemonSliceAPI.join_meeting() similarly omits bot_name from the API payload (api.py:162-163). If the LemonSlice backend assigns a different default display name than "LemonSlice Avatar", the chat relay will fail to filter the bot's own messages, causing the agent to respond to its own chat output in a feedback loop.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
If the LemonSlice backend assigns a different default display name than "LemonSlice Avatar"
It doesn't. We also use LemonSlice Avatar as the default if no name is provided.
3 participants
Adds third-party meeting support (Zoom, Google Meet, Teams, Webex) to the LemonSlice avatar plugin.
AvatarSession.join_meeting()sends the avatar into a meeting via the LemonSlice API; the plugin then connects to a LemonSlice-managed relay WebSocket that streams mixed meeting audio into MeetingAudioInput for STT (bypassing LiveKit room audio). Avatar speech is published back into the meeting through the existing LiveKit avatar path. Optionally, meeting chat messages can be relayed into the agent session as user input (listen_to_meeting_chat).Relevant LemonSlice endpoints are documented here and here
Adds new agent worker script in
examplesto demonstrate usage