Add livekit-plugins-funasr (FunASR/SenseVoice local STT)

[LauraGPT]

This PR adds livekit-plugins-funasr, a fully local speech-to-text plugin backed by FunASR and SenseVoice.

Why

SenseVoice is a non-autoregressive multilingual ASR model with strong Chinese and Cantonese support. It runs on-device without an API key and gives LiveKit agents a local alternative for Chinese, Cantonese, English, Japanese, and Korean speech.

What

• Adds FunASRSTT as a non-streaming LiveKit STT implementation.
• Combines incoming audio, resamples it to 16 kHz, converts stereo to mono, strips SenseVoice rich tags, and returns the final transcript plus detected language.
• Defers model initialization to the first recognition and runs both loading and inference off the event loop.
• Keeps lazy initialization and model.generate serialized in the worker thread, including when an awaiting request is cancelled.
• Implements the standard Plugin.download_files() hook so the default SenseVoice model can be prefetched before agent startup.
• Supports configurable model, device, language, and inverse text normalization.
• Registers the package in the root workspace, livekit-agents[funasr], and uv.lock with complete PyTorch and torchaudio runtime dependencies.
• Covers package imports, lazy loading, cancellation safety, model prefetch, resampling, and non-retryable local failures.

Usage

from livekit.plugins import funasr

stt = funasr.STT(model="iic/SenseVoiceSmall")

Prefetch the default model before starting an agent:

python -m livekit.agents download-files

CPU is the safe default. For GPU inference, install PyTorch and torchaudio builds matching the host CUDA runtime and pass device="cuda".

Verification

Validated on exact signed head 38d2558fda57ff464b18346d91c1db650fabeb0e, synchronized with livekit/agents@b634d4cc40d4be6cd15fa4b03ac9c11d65c24522 and 0 commits behind main:

• GitHub: 12 successful, 3 expected skips, 0 failed or pending checks; Devin Review passes and all 10 review threads are resolved
• focused FunASR plugin suite: 6 passed under the locked minimum runtime, funasr==1.3.16
• uv lock --check, Ruff format/check, mypy, and git diff --check pass
• the plugin version and livekit-agents[funasr] extra are aligned at 1.6.8
• the real download_files() hook downloads and loads the default iic/SenseVoiceSmall model with all checkpoint keys matched
• real first-request lazy loading plus CPU inference transcribes the 5.547-second official sample exactly as 欢迎大家来体验达摩院推出的语音识别模型。 and reports language zh
• wheel and sdist build successfully; the wheel contains only the FunASR plugin code, type marker, and package metadata

[LauraGPT]

Thanks for the review! Addressed the findings:

• Added the py.typed marker (PEP 561), so the package is type-checkable.
• model property now returns the configured model id instead of a hardcoded string.
• Excluded the nospeech classification label from the reported detected language (it is not a language).
• Serialized model.generate with an asyncio.Lock for thread-safety across concurrent recognitions sharing one instance.

CI is green (ruff + type-check). The transcription core (16-bit PCM -> resample to 16k -> SenseVoice -> cleaned text) was verified locally.

[LauraGPT]

Fresh FunASR/SenseVoice-side validation recheck (2026-07-07 UTC):

• PR is open, non-draft, and mergeable at head 348c46f; mergeable_state=blocked is currently the review gate.
• Thread-aware review audit shows 6 total review threads and 0 current unresolved threads. The previous AudioResampler(..., num_channels=...) discussion is resolved.
• GitHub checks visible on the head are green/skipped as expected: ruff, type-check (3.10), type-check (3.13), unit-tests, platform blockguard tests, release gate, Devin Review, and license/cla are successful; matrix/evaluation plugin jobs are skipped.
• Local focused validation on the current PR checkout:
  • git diff --check upstream/main...HEAD passed.
  • python3 -m py_compile livekit-plugins/livekit-plugins-funasr/livekit/plugins/funasr/stt.py livekit-plugins/livekit-plugins-funasr/livekit/plugins/funasr/__init__.py tests/test_plugin_funasr_stt.py passed.
  • livekit-plugins/livekit-plugins-funasr/pyproject.toml parses; package name is livekit-plugins-funasr with dependencies livekit-agents>=1.2.0, funasr>=1.1.0, and numpy.

Caveat: I did not rerun the full local pytest target in the bare ops checkout because the monorepo runtime package dependencies (livekit.api etc.) are not installed there; GitHub's own unit/type/ruff checks have already run successfully on this head.

[LauraGPT]

Updated this PR to current main and completed the packaging/runtime pass at afab30a:

• the branch is 0 commits behind and the plugin now follows the 1.6.5 release line;
• added livekit-agents[funasr], a regenerated uv.lock, complete PyTorch/torchaudio runtime dependencies, and a real package-import regression;
• local verification passes: 3 FunASR tests, full Ruff/format, mypy across 622 files, wheel/sdist build, and standalone Python 3.10 dependency resolution;
• real SenseVoiceSmall inference through STT.recognize returns the expected Chinese transcript;
• GitHub CI is complete with 9 successes, 3 expected fork skips, and no failures or pending jobs.

@tinalenguyen, would you be able to review the new local STT plugin when convenient?

[LauraGPT]

Synced this PR to current main; exact head cf327cc3c53ac1fad921fd746f745660152e6d99 is 0 commits behind and mergeable. The merge changed none of the FunASR implementation or test files.

Fresh local verification passes the 3 focused plugin regressions, Ruff and format checks, full mypy across 622 source files, uv lock --check, wheel/sdist build, and git diff --check. GitHub has also completed with all visible checks green (plus the expected fork skips) and no unresolved review threads.

@tinalenguyen, could you review the local FunASR/SenseVoice STT plugin when convenient?

[LauraGPT]

Refreshed this PR on top of current livekit/agents@e731eac and pushed head ef94cb81475c350ad6daf5119426d72082b27920.

What changed in the refresh:

• resolved the livekit-agents/pyproject.toml optional-dependency conflict by keeping the upstream >=1.6.6 plugin baseline and preserving the new funasr extra
• aligned livekit-plugins-funasr with the current release baseline: livekit-agents>=1.6.6, funasr>=1.3.16, and numba>=0.61.0 so the lock does not resolve an old Python-incompatible numba stack
• regenerated uv.lock; uv lock --check now passes

Local validation on the refreshed head:

• python -m pytest tests/test_plugin_funasr_stt.py -q -> 3 passed
• python -m pytest tests/test_plugin_funasr_stt.py tests/test_stt_context.py tests/test_inference_stt_context.py tests/test_plugin_assemblyai_stt.py -q -> 157 passed
• ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• git diff --check

GitHub now reports the PR mergeable again; repository CI is running on the refreshed head.

[LauraGPT]

Current-head verification for e07dd96bae706670f38ac3353127e0c2168e93d7:

• Rechecked the Devin threads with GraphQL: all review threads are resolved. The latest version-mismatch finding is fixed: livekit-agents[funasr] now requires livekit-plugins-funasr>=1.6.6, and livekit/plugins/funasr/version.py reports __version__ = "1.6.6".
• The py.typed marker exists, and local inference failures are raised as APIConnectionError(..., retryable=False).
• GitHub check rollup currently has no bad or pending checks; the PR is mergeable and only waiting for review.

Fresh focused validation on ind-gpu8:

.venv/bin/python -m pytest tests/test_plugin_funasr_stt.py -q
# 3 passed, 4 warnings

.venv/bin/ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# All checks passed!

.venv/bin/ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# 5 files already formatted

/cpfs_speech/user/zhifu.gzf/venv_uv_bootstrap/bin/uv lock --check
# exit 0, resolved 359 packages

python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
git diff --check upstream/main...HEAD

[LauraGPT]

Current-head verification for 9f995f7ee5fc4aa1a7085f49f97270217a54deac:

• GitHub check rollup is green/skipped only; the PR is mergeable and review-gated.
• Local focused plugin verification on ind-gpu8 passes:

.venv/bin/python -m pytest tests/test_plugin_funasr_stt.py -q
# 3 passed, 4 warnings

.venv/bin/ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# All checks passed!

.venv/bin/ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# 5 files already formatted

/cpfs_speech/user/zhifu.gzf/venv_uv_bootstrap/bin/uv lock --check
# resolved 364 packages, exit 0

python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
git diff --check upstream/main...HEAD

No new FunASR/SenseVoice-side blocker from this refresh.

[LauraGPT]

Hi @tinalenguyen — this new local FunASR/SenseVoice STT plugin is current with main and ready for plugin-owner review. On exact head 8cc481cbba89469cb9df9ac8d58e23ac995b8d06, all 11 GitHub checks pass; the focused suite is 3/3 under the locked minimum funasr==1.3.16; Ruff, lock, and diff checks pass; and a real 5.547-second CPU sample returns the expected Chinese transcript with language zh. I also refreshed the PR body with the exact validation evidence. Would you be able to take a look when convenient?

[LauraGPT]

@livekit/agent-devs, this FunASR/SenseVoice plugin is current with main and ready for maintainer review on signed head 38d2558fda57ff464b18346d91c1db650fabeb0e.

The latest review cycle is complete: model loading now runs off the event loop, the default model can be prefetched through download_files(), public APIs are documented, and worker-thread serialization remains intact after caller cancellation. All 10 review threads are resolved.

Current evidence:

• 0 commits behind main; GitHub reports the PR mergeable
• 12 successful checks, 3 expected skips, 0 failures or pending checks, including Devin Review
• 6 focused plugin tests, Ruff, mypy, lock check, and diff check pass locally
• real first-request lazy loading returns the exact expected Chinese transcript with language zh
• wheel and sdist build successfully at version 1.6.8

The PR body now contains the exact current-head validation details.