Add CTC zh-CN Mandarin ASR with THCHS-30 benchmarking by Alex-Wengg · Pull Request #475 · FluidInference/FluidAudio

Alex-Wengg · 2026-04-02T23:00:15Z

Summary

This PR adds complete Mandarin Chinese (zh-CN) ASR support using Parakeet CTC 0.6B model with comprehensive benchmarking infrastructure.

Changes

CTC zh-CN ASR Integration

Add CtcZhCnManager for Mandarin Chinese speech recognition
Int8 quantized encoder (571 MB) with greedy CTC decoding
CLI command: fluidaudiocli ctc-zh-cn-transcribe

THCHS-30 Benchmark Pipeline

GitHub Actions workflow for automated CI benchmarking
Swift CLI benchmark with HuggingFace auto-download support
Python benchmark scripts for alternative testing
Dataset: FluidInference/THCHS-30-tests

Benchmark Results (100 samples)

Mean CER: 8.37%
Median CER: 6.67%
Mean Latency: 1,284 ms
Distribution: 69% of samples achieve <10% CER

Comparison

Dataset	Mean CER	Notes
THCHS-30	8.37%	Clean Chinese corpus
FLEURS zh_cn	10.22%	Some English contamination

THCHS-30 shows 18% better CER due to cleaner data quality.

Testing

Run benchmark locally:

# Swift CLI (auto-downloads dataset)
swift run -c release fluidaudiocli ctc-zh-cn-benchmark --auto-download --samples 100

# Python script
python Scripts/benchmark_ctc_zh_cn.py --max-samples 100

CI Integration

GitHub Actions automatically:

Downloads THCHS-30 from HuggingFace
Runs 100-sample benchmark
Validates CER < 10% threshold
Posts results as PR comment

Files Added

.github/workflows/ctc-zh-cn-benchmark.yml - CI workflow
Sources/FluidAudio/ASR/CtcZhCnManager.swift - Core manager
Sources/FluidAudioCLI/Commands/ASR/CtcZhCnBenchmark.swift - Benchmark command
Scripts/benchmark_ctc_zh_cn.py - Python benchmark
Scripts/test_ctc_zh_cn_hf.py - HF dataset test

Model Info

Model: parakeet-ctc-0.6b-zh-cn
Encoder: Int8 quantized (571 MB)
Decoder: CTC greedy decoding
Platform: Apple Silicon (M-series)
Dataset: THCHS-30 (2,495 test utterances, 10 speakers)

🤖 Generated with Claude Code

Fixes actor isolation violations that appeared with stricter Swift 6 concurrency checking in newer Xcode versions. The issue was caused by extracting actor references from properties into local variables using if-let/guard-let, which changes isolation context and risks data races. Solution uses optional chaining with proper scoping: - Avoids force unwrapping (repository rule) - Prevents actor isolation violations (Swift 6 requirement) - Handles actor reentrancy safely (asrManager can become nil after await) - Uses if-let for conditional blocks to avoid skipping critical state updates Changes: - reset(): Optional chaining for resetDecoderState - finish(): Guard-let on processTranscriptionResult return value - processWindow(): Guard-let for required results, if-let for optional rescoring - All early-return guards use guard-let at function level - Conditional block uses if-let to avoid premature function exit Fixes prevent partial state mutations and ensure subscriber notifications always occur even if optional vocabulary rescoring fails.

Moves state mutations to occur AFTER all required async calls complete, preventing inconsistent state if asrManager becomes nil during suspension. Previously, if the second guard-let failed (line 408), the function would return after having already mutated: - accumulatedTokens - lastProcessedFrame - segmentIndex - processedChunks This created inconsistency where tokens were accumulated but transcript state and subscriber notifications were skipped. Solution: Delay all state mutations until after both required async calls (transcribeChunk and processTranscriptionResult) complete successfully.

## Bug Fix Fixed critical bug in decoder projection normalization that caused 82-113% WER (complete model failure). The issue was in TdtModelInference.swift where the destination stride was hardcoded to 1 instead of using the actual MLMultiArray stride, causing incorrect BLAS copy operations. **Impact**: All TDT models (v2, v3, tdt-ctc-110m) were producing garbage output **Root cause**: Hardcoded stride in normalizeDecoderProjection() **Fix**: Use actual destination array stride from MLMultiArray ## Refactoring Extracted reusable decoder components into separate files for better maintainability and code organization: - TdtModelInference.swift: Centralized model inference operations - runDecoder(): LSTM decoder execution - runJointPrepared(): Joint network execution with zero-copy optimization - normalizeDecoderProjection(): BLAS-based projection normalization (BUG FIX HERE) - TdtJointDecision.swift: Joint network decision data structure - TdtJointInputProvider.swift: Reusable feature provider for joint network - TdtDurationMapping.swift: Duration bin mapping utilities - TdtFrameNavigation.swift: Frame position calculation for streaming Simplified TdtDecoderV3.swift from 700+ lines to ~500 lines by extracting common operations. ## Validation Full test-clean benchmark (2,620 files): - Parakeet v3: WER 2.64% (baseline: 2.6%) ✓ - Parakeet v2: WER 3.79% (baseline: 3.8%) ✓ - TDT-CTC-110M: WER 3.56% (baseline: 3.6%) ✓ - All models: No regressions, performance matches baselines Perfect transcriptions: 74.3% (1,947/2,620 files) Processing speed: 45x real-time (5.4 hours audio in 7.2 minutes)

Added documentation for the new refactored decoder components: - TdtModelInference.swift - TdtJointDecision.swift - TdtJointInputProvider.swift - TdtDurationMapping.swift - TdtFrameNavigation.swift These files were extracted from TdtDecoderV3.swift as part of the decoder refactoring to improve code organization and maintainability.

- Remove private makeBlasIndex that shadowed global version - Flatten nested conditionals in TdtFrameNavigation and TdtDecoderV3 - Add comprehensive unit tests for refactored TDT components (30 tests) All tests pass (30/30). Global makeBlasIndex supports negative strides for reverse traversal, which the private version blocked. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Integrates Parakeet CTC 0.6B zh-CN model for Mandarin Chinese speech recognition. - Add CtcZhCnManager for full pipeline transcription (preprocessor → encoder → CTC decoder) - Add CtcZhCnModels for model loading from HuggingFace - Support int8 (0.55GB) and fp32 (1.1GB) encoder variants - Add ctc-zh-cn-transcribe CLI command - Add ctc-zh-cn-benchmark CLI command (placeholder) - Greedy CTC decoding with proper blank/repeat handling - 10.22% CER on FLEURS Mandarin Chinese (100 samples) Performance: - Mean CER: 10.22% (matches Python baseline: 10.45%) - 46% of samples < 5% CER (near perfect) - Auto-download from HuggingFace on first use Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

- Add GitHub Actions workflow for CI benchmarking - Implement THCHS-30 dataset auto-download from HuggingFace - Add Swift CLI benchmark command with local/remote dataset support - Add Python benchmark scripts for alternative testing - Expected performance: 8.37% mean CER (100 samples) Dataset: FluidInference/THCHS-30-tests Model: parakeet-ctc-0.6b-zh-cn (int8, 571 MB)

github-actions · 2026-04-02T23:01:12Z

CTC zh-CN Benchmark Results ❌

Status: Benchmark failed (see logs)

THCHS-30 (Mandarin Chinese)

Metric	Value	Target	Status
Mean CER	%	<10%	⚠️
Median CER	%	<7%	⚠️
Mean Latency	ms	-	-
Samples		100	⚠️

CER Distribution

Range	Count	Percentage
<5%		NaN%
<10%		NaN%
<20%		NaN%

_{Model: parakeet-ctc-0.6b-zh-cn (int8, 571 MB) • Dataset: THCHS-30 (Tsinghua University)}
_{Test runtime: • 04/02/2026, 07:01 PM EST}

_{CER = Character Error Rate • Lower is better • Calculated using Levenshtein distance with normalized text}

github-actions · 2026-04-02T23:01:57Z

VAD Benchmark Results

❌ Benchmark failed - no results generated

github-actions · 2026-04-02T23:02:00Z

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Optimal clustering with Hungarian algorithm for maximum accuracy

Metric	Value	Target	Status	Description
DER	NaN%	<20%	⚠️	Diarization Error Rate (lower is better)
RTFx	NaNx	>1.0x	⚠️	Real-Time Factor (higher is faster)

Offline VBx Pipeline Timing Breakdown

Time spent in each stage of batch diarization

Stage	Time (s)	%	Description
Model Download	NaN	NaN	Fetching diarization models
Model Compile	NaN	NaN	CoreML compilation
Audio Load	NaN	NaN	Loading audio file
Segmentation	NaN	NaN	VAD + speech detection
Embedding	NaN	NaN	Speaker embedding extraction
Clustering (VBx)	NaN	NaN	Hungarian algorithm + VBx clustering
Total	NaN	100	Full VBx pipeline

Speaker Diarization Research Comparison

Offline VBx achieves competitive accuracy with batch processing

Method	DER	Mode	Description
FluidAudio (Offline)	NaN%	VBx Batch	On-device CoreML with optimal clustering
FluidAudio (Streaming)	17.7%	Chunk-based	First-occurrence speaker mapping
Research baseline	18-30%	Various	Standard dataset performance

Pipeline Details:

Mode: Offline VBx with Hungarian algorithm for optimal speaker-to-cluster assignment
Segmentation: VAD-based voice activity detection
Embeddings: WeSpeaker-compatible speaker embeddings
Clustering: PowerSet with VBx refinement
Accuracy: Higher than streaming due to optimal post-hoc mapping

_{🎯 Offline VBx Test • AMI Corpus ES2004a • NaNs meeting audio • NaNs processing • Test runtime: N/A • 04/02/2026, 07:01 PM EST}

github-actions · 2026-04-02T23:03:25Z

ASR Benchmark Results ⚠️

Status: Some benchmarks failed (see logs)

Parakeet v3 (multilingual)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	%	%	x	⚠️
test-other	%	%	x	⚠️

Parakeet v2 (English-optimized)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	%	%	x	⚠️
test-other	%	%	x	⚠️

Streaming (v3)

Metric	Value	Description
WER	%	Word Error Rate in streaming mode
RTFx	x	Streaming real-time factor
Avg Chunk Time	s	Average time to process each chunk
Max Chunk Time	s	Maximum chunk processing time
First Token	s	Latency to first transcription token
Total Chunks		Number of chunks processed

Streaming (v2)

Metric	Value	Description
WER	%	Word Error Rate in streaming mode
RTFx	x	Streaming real-time factor
Avg Chunk Time	s	Average time to process each chunk
Max Chunk Time	s	Maximum chunk processing time
First Token	s	Latency to first transcription token
Total Chunks		Number of chunks processed

_{Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming}

_{files per dataset • Test runtime: • 04/02/2026, 07:03 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

_{Testing methodology follows HuggingFace Open ASR Leaderboard}

github-actions · 2026-04-02T23:03:42Z

Qwen3-ASR int8 Smoke Test ❌

Check	Result
Build	✅
Model download	❌
Model load	❌
Transcription pipeline	❌
Decoder size	571 MB (vs 1.1 GB f32)

Performance Metrics

Metric	CI Value	Expected on Apple Silicon
Median RTFx	x	~2.5x
Overall RTFx	x	~2.5x

_Runtime:

_{Note: CI VM lacks physical GPU — CoreML MLState (macOS 15) KV cache produces degraded results on virtualized runners. On Apple Silicon: ~1.3% WER / 2.5x RTFx.}

github-actions · 2026-04-02T23:04:01Z

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric	Value	Target	Status	Description
DER	NaN%	<30%	⚠️	Diarization Error Rate (lower is better)
JER	NaN%	<25%	⚠️	Jaccard Error Rate
RTFx	NaNx	>1.0x	⚠️	Real-Time Factor (higher is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage	Time (s)	%	Description
Model Download	NaN	NaN	Fetching diarization models
Model Compile	NaN	NaN	CoreML compilation
Audio Load	NaN	NaN	Loading audio file
Segmentation	NaN	NaN	Detecting speech regions
Embedding	NaN	NaN	Extracting speaker voices
Clustering	NaN	NaN	Grouping same speakers
Total	NaN	100	Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method	DER	Notes
FluidAudio	NaN%	On-device CoreML
Research baseline	18-30%	Standard dataset performance

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

M2 MacBook Air (2022): Runs at 150 RTFx real-time
Performance scales with Apple Neural Engine capabilities

_{🎯 Speaker Diarization Test • AMI Corpus ES2004a • NaNs meeting audio • NaNs diarization time • Test runtime: N/A • 04/02/2026, 07:04 PM EST}

github-actions · 2026-04-02T23:05:26Z

Parakeet EOU Benchmark Results ❌

Status: Benchmark failed (see logs)
Chunk Size: ms
Files Tested: /

Performance Metrics

Metric	Value	Description
WER (Avg)	%	Average Word Error Rate
WER (Med)	%	Median Word Error Rate
RTFx	x	Real-time factor (higher = faster)
Total Audio	s	Total audio duration processed
Total Time	s	Total processing time

Streaming Metrics

Metric	Value	Description
Avg Chunk Time	s	Average chunk processing time
Max Chunk Time	s	Maximum chunk processing time
EOU Detections		Total End-of-Utterance detections

_{Test runtime: • 04/02/2026, 07:05 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O}

github-actions · 2026-04-02T23:05:29Z

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Metric	Value	Target	Status
DER	0.0%	<35%	✅
Miss Rate	0.0%	-	-
False Alarm	0.0%	-	-
Speaker Error	0.0%	-	-
RTFx	0.0x	>1.0x	⚠️
Speakers	0/0	-	-

_{Sortformer High-Latency • ES2004a • Runtime: N/A • 2026-04-02T23:05:27.632Z}

Alex-Wengg · 2026-04-03T01:45:46Z

Closing this PR in favor of a clean rebase: the new PR will be created from branch fix/swift6-concurrency-slidingwindow-rebased which is rebased off main without the conflicting decoder refactoring commit.

## Summary This PR adds **experimental** Mandarin Chinese ASR support via the CTC zh-CN model and includes critical Swift 6 concurrency fixes for `SlidingWindowAsrManager`. > **⚠️ Experimental Feature**: CTC zh-CN Mandarin ASR is an early preview. The API and performance characteristics may change in future releases. ## Swift 6 Concurrency Fixes ### Fixed Issues - **Removed premature state mutations** in `processWindow()` that violated Swift 6 actor isolation - State updates (`accumulatedTokens`, `lastProcessedFrame`, `segmentIndex`, `processedChunks`) now occur **after** all async calls complete successfully - Prevents data races when async calls fail mid-execution ### Changes - `SlidingWindowAsrManager.processWindow()`: Moved state mutation to after async guard statements - Ensures atomic state updates only when processing succeeds ## CTC zh-CN Mandarin ASR Integration (Experimental) ### New Features #### Models - **CtcZhCnManager**: High-level API for Mandarin Chinese ASR using CTC decoder - **CtcZhCnModels**: Model management with int8/fp32 encoder variants - Int8: 571 MB (default) - FP32: 1.1 GB - Auto-downloads from HuggingFace: `FluidInference/parakeet-ctc-0.6b-zh-cn-coreml` #### CLI Commands ```bash # Transcribe Mandarin audio swift run fluidaudiocli ctc-zh-cn-transcribe audio.wav # Benchmark on THCHS-30 dataset (full 2,495 samples) swift run fluidaudiocli ctc-zh-cn-benchmark --auto-download # Benchmark subset (100 samples for faster testing) swift run fluidaudiocli ctc-zh-cn-benchmark --auto-download --samples 100 ``` #### Benchmark Results (THCHS-30 Full Test Set) **Full dataset** (2,495 samples): - **Mean CER**: 8.23% - **Median CER**: 6.45% - **CER = 0% (perfect)**: 435 samples (17.4%) - **Distribution**: 67.1% of samples <10% CER, 93.2% <20% CER - **Mean Latency**: 614 ms - **Mean RTFx**: 14.83x ### Dataset **THCHS-30** - Mandarin Chinese speech corpus from Tsinghua University - 30 hours of clean speech - 50 speakers - 2,495 test utterances (10 speakers, 250 unique sentences) - Content domain: News (not classical literature) - Source: http://www.openslr.org/18/ - HuggingFace: `FluidInference/THCHS-30-tests` ### Text Normalization CER calculation includes: - Chinese punctuation removal (，。！？、；：\u{201C}\u{201D}\u{2018}\u{2019}) - English punctuation removal (,.!?;:()[]{}\\<>"'-) - Arabic digit → Chinese character conversion (0→零, 1→一, etc.) - Whitespace normalization - Levenshtein distance calculation ## Devin Review Fixes ✅ Addressed all issues from [Devin code review](https://app.devin.ai/review/fluidinference/fluidaudio/pull/476): ### Review #1 (4 issues) 1. **✅ Fixed digit-to-Chinese conversion** - Added missing normalization (0→零, 1→一, etc.) that was inflating CER by ~1.66% 2. **✅ Added unit tests** - Created 13 comprehensive test cases for text normalization, CER calculation, and Levenshtein distance 3. **✅ Fixed CI dataset cache path** - Not applicable after CI workflow removal 4. **✅ Fixed CI model cache path** - Not applicable after CI workflow removal ### Review #2 (2 issues) 5. **✅ Fixed CER threshold mismatch** - Not applicable after CI workflow removal 6. **✅ Fixed saveResults NaN crash** - Added guard for empty results array to prevent division by zero ### Review #3 (2 issues) 7. **✅ Fixed FP32 encoder download** - Include both int8 and fp32 encoders in `requiredModels` set 8. **✅ Fixed AsrManager CTC-only handling** - Throw explicit error instead of routing to incompatible TDT decoder ### Additional Fixes - **✅ Fixed Unicode curly quotes** - Used escape sequences (`\u{201C}` etc.) in both source and tests - Added missing English punctuation removal - Added missing Chinese quotation mark handling ## Files Changed ### Swift 6 Concurrency - `Sources/FluidAudio/ASR/Parakeet/SlidingWindow/SlidingWindowAsrManager.swift` - `Sources/FluidAudio/ASR/Parakeet/AsrManager.swift` (added .ctcZhCn case + error handling) ### CTC zh-CN Integration - `Sources/FluidAudio/ASR/Parakeet/CtcZhCnManager.swift` (new) - `Sources/FluidAudio/ASR/Parakeet/CtcZhCnModels.swift` (new) - `Sources/FluidAudioCLI/Commands/ASR/CtcZhCnTranscribeCommand.swift` (new) - `Sources/FluidAudioCLI/Commands/ASR/CtcZhCnBenchmark.swift` (new) - `Sources/FluidAudio/ModelNames.swift` (updated - both encoder variants) - `Documentation/Benchmarks.md` (updated - marked experimental) ### Tests - `Tests/FluidAudioTests/ASR/Parakeet/CtcZhCnTests.swift` (new - 13 test cases) ## Testing - [x] Swift 6 concurrency fixes pass existing tests - [x] CTC zh-CN transcription tested manually - [x] THCHS-30 full benchmark: 8.23% mean CER (2,495 samples) - [x] Unit tests: 13 test cases for normalization and CER (100% passing) - [x] Text normalization matches baseline exactly - [x] FP32 encoder download verified ## Notes - This PR is a clean rebase of #475 off main - Skipped conflicting decoder refactoring commit (superseded by #474) - **Experimental feature**: CTC zh-CN API may change in future releases - **No CI workflow**: Benchmarks are run manually for experimental features

Alex-Wengg and others added 8 commits March 30, 2026 13:21

Merge main into fix/swift6-concurrency-slidingwindow

884ffba

Alex-Wengg closed this Apr 3, 2026

Alex-Wengg mentioned this pull request Apr 3, 2026

Add experimental CTC zh-CN Mandarin ASR #476

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CTC zh-CN Mandarin ASR with THCHS-30 benchmarking#475

Add CTC zh-CN Mandarin ASR with THCHS-30 benchmarking#475
Alex-Wengg wants to merge 8 commits intomainfrom
fix/swift6-concurrency-slidingwindow

Alex-Wengg commented Apr 2, 2026 •

edited by devin-ai-integration bot

Loading

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Alex-Wengg commented Apr 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Alex-Wengg commented Apr 2, 2026 • edited by devin-ai-integration bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

CTC zh-CN ASR Integration

THCHS-30 Benchmark Pipeline

Benchmark Results (100 samples)

Comparison

Testing

CI Integration

Files Added

Model Info

Uh oh!

github-actions bot commented Apr 2, 2026

CTC zh-CN Benchmark Results ❌

THCHS-30 (Mandarin Chinese)

CER Distribution

Uh oh!

github-actions bot commented Apr 2, 2026

VAD Benchmark Results

Uh oh!

github-actions bot commented Apr 2, 2026

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Offline VBx Pipeline Timing Breakdown

Speaker Diarization Research Comparison

Uh oh!

github-actions bot commented Apr 2, 2026

ASR Benchmark Results ⚠️

Parakeet v3 (multilingual)

Parakeet v2 (English-optimized)

Streaming (v3)

Streaming (v2)

Expected RTFx Performance on Physical M1 Hardware:

Uh oh!

github-actions bot commented Apr 2, 2026

Qwen3-ASR int8 Smoke Test ❌

Performance Metrics

Uh oh!

github-actions bot commented Apr 2, 2026

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Diarization Pipeline Timing Breakdown

Speaker Diarization Research Comparison

Uh oh!

github-actions bot commented Apr 2, 2026

Parakeet EOU Benchmark Results ❌

Performance Metrics

Streaming Metrics

Uh oh!

github-actions bot commented Apr 2, 2026

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Uh oh!

Alex-Wengg commented Apr 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Alex-Wengg commented Apr 2, 2026 •

edited by devin-ai-integration bot

Loading