Skip to content

feat(audio): switch playback to <audio src=…> via /api/calls/:id/audio#27

Merged
revtex merged 1 commit intodevfrom
feat/audio-element-playback
Apr 25, 2026
Merged

feat(audio): switch playback to <audio src=…> via /api/calls/:id/audio#27
revtex merged 1 commit intodevfrom
feat/audio-element-playback

Conversation

@revtex
Copy link
Copy Markdown
Owner

@revtex revtex commented Apr 25, 2026

Summary

Switch the SPA audio pipeline from base64-over-WebSocket + Web Audio decodeAudioData to a single persistent hidden HTMLAudioElement that streams each call from the existing GET /api/calls/:id/audio endpoint, authenticated via the os_session cookie introduced in #25.

Pairs with #26 — merge order matters

This PR depends on backend PR #26 ("drop embedded audio from CAL messages"). Merge #26 first, then this immediately. Either side works on its own (the frontend simply ignores any base64 audio field the backend still sends, and the backend with this frontend just stops shipping bytes nobody reads), but the clean cut is the goal.

User-visible fix

Mobile Edge on Android now plays AAC calls. Edge ships AAC support in HTMLMediaElement (via Android MediaCodec) but not in Web Audio's decodeAudioData — the old path threw EncodingError and the existing fallback was hidden behind a .catch(() => {}), so calls silently never played. Routing every call through the platform <audio> element fixes this without touching the gain graph the volume slider depends on.

Side benefit: the client no longer holds a base64 string + decoded AudioBuffer per call in memory.

Architecture

  • One persistent hidden HTMLAudioElement is created during the existing first-user-gesture audio-unlock and wired through MediaElementAudioSourceNode → GainNode → AudioContext.destination. The volume slider, mute, and beep all continue to flow through the same gain node.
  • The WS client no longer holds an audio callback. On CAL it dispatches callReceived(call) to Redux.
  • A new audioListenerMiddleware subscribes to the calls slice and drives the player.
  • Per call: audio.src = "/api/calls/" + call.id + "/audio", preload = "auto", play() on canplay; ended/error advance the queue.
  • Autoplay-unlock additionally calls audio.play().then(pause).catch(noop) on the persistent element inside the user-gesture handler so subsequent programmatic play() calls succeed on mobile.

Dropped code paths

  • decodeAudioData and the in-memory AudioBuffer queue.
  • The base64ToArrayBuffer helper and the audio field parse in the WS client.
  • The audio Blob allocation and URL.createObjectURL for playback.
  • The playViaAudioElement Web-Audio-failure fallback (now the only path).
  • The "Suppress EncodingError on Mobile Edge" .catch(() => {}) hack.

Download / share

  • BookmarksPanel and SearchPanel download buttons are now plain <a href="/api/calls/:id/audio" download="…"> anchors. The browser handles the streaming download with the cookie attached automatically.
  • SharedCall page uses <audio src="/api/shared/:token/audio"> directly.

Verification

  • pnpm exec tsc --noEmit clean.
  • pnpm test --run — 188 / 188 pass across 20 suites.
  • rg 'decodeAudioData|base64ToArrayBuffer' frontend/src — 0 hits.
  • rg 'createObjectURL.*[Aa]udio|audioCallback|setAudioCallback' frontend/src — 0 hits.
  • rg 'playViaAudioElement|EncodingError' frontend/src — 0 hits.

Self-review

  • No XSS surface added. No dangerouslySetInnerHTML. URLs are constructed from numeric call.id / opaque share tokens.
  • No token leakage. The access JWT continues to live in Redux memory only. The os_session cookie is httpOnly; the SPA never reads or writes it. No bearer header is added to the <audio> request — the cookie does the work.
  • publicAccess=true deployments still work. The server's OptionalJWTOrSessionAuth falls through to anonymous when no auth is presented, and the <audio> element will simply omit the cookie for unauthenticated sessions, which is exactly what public access expects.
  • Mute / volume continue to flow through the existing GainNode, so any change to the volume slider during playback applies in real time.

Files

8 files changed, +277 / −426. Net deletion: 149 LOC.

- Replace base64-over-WS + Web Audio decode pipeline with a single
  persistent hidden HTMLAudioElement, wired through
  MediaElementAudioSourceNode → GainNode → AudioContext.destination so
  the existing volume slider and gain graph keep working.
- WS client no longer parses or holds an audio callback; on CAL it
  dispatches callReceived(call) to Redux. A new listener middleware
  drives the player off the calls slice.
- Per-call playback flow: audio.src = /api/calls/:id/audio, preload
  auto, play() on canplay, advance queue on ended/error.
- Download buttons in BookmarksPanel and SearchPanel are now plain
  <a download> anchors pointing at the same authenticated endpoint.
- Shared-call page uses <audio src=/api/shared/:token/audio> directly.
- Autoplay-unlock now also primes the persistent <audio> element
  inside the user-gesture handler (play().then(pause)) so subsequent
  programmatic play() calls succeed on Mobile Edge / Mobile Safari.
- Tests updated to drive the new flow; 188 frontend tests pass.

Pairs with backend PR #26 dropping audioData from CAL frames; merge
that one first, then this.
@revtex revtex merged commit 37619be into dev Apr 25, 2026
7 checks passed
@revtex revtex deleted the feat/audio-element-playback branch April 25, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant