docs: add LS-EEND vs Sortformer enrollment feedback from integration testing#397
Conversation
…testing Captures real-world integration feedback on speaker enrollment behavior: - LS-EEND slot collision with similar voices, score range limitations - Sortformer's stronger auto-mapping for pre-enrolled speaker workflows - Training data differences explaining the enrollment accuracy gap
|
Claude finished @Alex-Wengg's task —— View job PR Review Complete ✅Todo List:
Documentation Review SummaryI've completed a comprehensive review of this PR's documentation changes. The additions provide valuable real-world integration insights about enrollment capabilities comparing LS-EEND vs Sortformer models. Key Findings:✅ Technical accuracy: Content correctly explains sigmoid bounds, training data differences, and architectural limitations Review Submitted:I've submitted a non-blocking review with positive feedback and minor suggestions for improvement. The documentation enhancements significantly improve user guidance for enrollment-based workflows. The PR is ready for merge with the suggested minor improvements being optional enhancements rather than blocking issues. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR adds valuable documentation about enrollment feedback comparing LS-EEND vs Sortformer based on real integration testing. The content is technically accurate, well-structured, and provides actionable insights for users choosing between models for enrollment workflows.
Key Strengths:
- Evidence-based content from actual 4-speaker integration testing
- Clear technical explanations (sigmoid bounds, training data differences, architectural limitations)
- Good cross-referencing between documents
- Helpful additions to the comparison table
Minor suggestions:
- Consider briefly defining "Pre-enrolled speaker mapping" for new users
- Ensure consistent formatting (bold usage) across documents
Overall, this enhances the documentation quality and will help users make informed decisions. The technical details about LS-EEND's 0.2-0.8 score bounds and Sortformer's real-world training advantages are particularly valuable.
PocketTTS Smoke Test ✅
Runtime: 0m39s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality may differ from Apple Silicon. |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 0m32s • 03/18/2026, 03:41 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
Qwen3-ASR int8 Smoke Test ✅
Runtime: 4m22s Note: CI VM lacks physical GPU — CoreML MLState (macOS 15) KV cache produces degraded results on virtualized runners. On Apple Silicon: ~1.3% WER / 2.5x RTFx. |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 9m24s • 03/18/2026, 03:48 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 251.3s processing • Test runtime: 4m 29s • 03/18/2026, 03:51 PM EST |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 43.0s diarization time • Test runtime: 5m 21s • 03/18/2026, 03:56 PM EST |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 7m 19s • 2026-03-18T20:02:34.549Z |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
1 participant
Summary
Source: Discord integration feedback from Adam Tow and model architecture clarifications from Gradient Descent (LS-EEND sigmoid score range, training data composition, attractor suppression as potential path forward).
Test plan