Skip to content

feat(desktop): add local voice dictation - #3759

Open
johnmatthewtennant wants to merge 13 commits into
mainfrom
jtennant/local-stt-dictation
Open

feat(desktop): add local voice dictation#3759
johnmatthewtennant wants to merge 13 commits into
mainfrom
jtennant/local-stt-dictation

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Context

Buzz already runs Parakeet locally for huddle transcription, but message composers cannot use it. This adds an experiment-gated dictation path that keeps microphone audio and transcription on-device.

Summary

Add local voice dictation to desktop message composers. Users can click the microphone or hold Cmd+D (Ctrl+D on non-Mac platforms), speak into the local Parakeet pipeline, and receive the final transcript in the current draft.

Changes

  • Reuse the existing Parakeet model, resampler, VAD, and recognizer through a session-scoped dictation pipeline.
  • Capture microphone PCM in an AudioWorklet and send bounded, ordered batches to Tauri.
  • Scope transcript and stop events to the active session and composer context so late results cannot enter another draft, reply, or edit.
  • Flush accepted audio and the final recognizer buffer before completing dictation.
  • Make one Send click stop recording, wait for the final transcript, and submit the completed draft once.
  • Cancel dictation when the composer context changes, becomes disabled, or unmounts.
  • Add an experiment-gated microphone control and Cmd+D or Ctrl+D push-to-talk shortcut.

Related issue

Related: #1511 and #2742.

Testing

  • Exact-build manual validation covered the experiment gate, idle and recording states, local transcript insertion, quick shortcut release, context switching, and one-click Send finalization.
  • One Send click stopped capture and produced exactly one message containing the final transcript.

Screenshots

Idle composer:

Idle voice dictation control

Recording:

Active voice dictation control

Reviewer-reproducible examples

From a fresh checkout of the PR, launch the native app:

git fetch origin pull/3759/head:pr-3759
git switch pr-3759
. ./bin/activate-hermit
just dev

Then:

  1. Open Settings, enable the Voice Dictation experiment, and wait for the local speech model to become available.
  2. Open a writable message composer. Click the microphone, speak a distinctive phrase, then click the control again. The phrase should appear in the current draft.
  3. Hold Cmd+D on macOS or Ctrl+D elsewhere, speak, and release quickly. Recording should stop even when the key is released before microphone setup finishes.
  4. Start from an empty composer and click Send once while recording. Capture should stop, the final words should be included, and exactly one message should be sent after transcription finishes.
  5. Start dictation, then switch channels, replies, or edit targets. No transcript from the prior context should appear in the new context.

Observed on the exact PR build: transcript text appeared in the active draft, quick shortcut release stopped capture, context changes rejected late transcript text, and one Send click produced one message with the final words included.

@johnmatthewtennant

Copy link
Copy Markdown
Contributor Author

🤖 Exact-build composer states.

Idle

The experiment-gated microphone control is available in a writable composer.

01-idle

Recording

The active control changes to a prominent stop state while local dictation is capturing audio.

02-recording

johnmatthewtennant added a commit that referenced this pull request Jul 30, 2026
@johnmatthewtennant
johnmatthewtennant marked this pull request as ready for review July 30, 2026 17:31
@johnmatthewtennant
johnmatthewtennant requested a review from a team as a code owner July 30, 2026 17:31
Base automatically changed from jtennant/pocket-tts-natural-segmentation to jtennant/pocket-tts-streaming July 31, 2026 11:08
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/pocket-tts-streaming branch from 4d18f5b to efbc63d Compare July 31, 2026 17:03
@johnmatthewtennant
johnmatthewtennant marked this pull request as draft July 31, 2026 18:37
johnmatthewtennant and others added 11 commits July 31, 2026 14:38
Co-authored-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Co-authored-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Co-authored-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Co-authored-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Co-authored-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Co-authored-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Co-authored-by: Kenny Lopez <klopez4212@gmail.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/local-stt-dictation branch from a083f05 to c831ebe Compare July 31, 2026 18:53
@johnmatthewtennant
johnmatthewtennant changed the base branch from jtennant/pocket-tts-streaming to main July 31, 2026 18:54
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
@klopez4212
klopez4212 marked this pull request as ready for review August 1, 2026 06:36

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e43ef004eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


try {
// 1. Start the native STT engine — returns the session ID used to tag events.
const sessionId = await invoke<number>("start_dictation");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Register the stop listener before starting the native session

When two mounted composers start dictation close together, the second native start can replace this session after start_dictation returns but before the dictation-state listener is registered below. The native forwarder then emits this session's one-shot stopped event with no listener present, yet this hook continues opening the microphone; subsequent session-scoped stops cannot affect the replacement engine, leaving this composer stuck in isTranscribing and unable to start dictation again until it is remounted. Register the state listener before creating the session, or otherwise reconcile native state after listener registration.

Useful? React with 👍 / 👎.

@Chessing234

Copy link
Copy Markdown
Contributor

on-device dictation via parakeet (instead of huddle→relay) is a nice split — session_id on the events is a good guard against stale transcripts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants