Commit 43bcec7
Add streaming multi-talker Parakeet RNN-T ASR model
Add NVIDIA NeMo `EncDecMultiTalkerRNNTBPEModel`
(multitalker-parakeet-streaming-0.6b-v1) as its own model type,
exported as three ONNX graphs matching the nemotron-speech-streaming
cache-aware streaming deployment contract:
- encoder: FastConformer with speaker-kernel injection and streaming
channel/time caches (causal dw_striding subsampling, bias-free
linears, LayerNorm conv, chunked-limited attention, rel-pos KV cache).
- decoder: RNN-T LSTM prediction network (2-layer ONNX LSTM with
PyTorch->ONNX gate reordering).
- joint: RNN-T joint network emitting per-(time, target) logits.
Adds MultiTalkerRNNTTask, the ParakeetMultiTalkerModel/Config classes
with a .nemo loader (build_parakeet_multitalker), unit tests (graph
build + ORT smoke run of all three graphs), and an integration test
verifying streaming parity against the NeMo PyTorch reference
(encoder 3.9e-7, decoder 1.1e-6, joint argmax match).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Mason Corey <masoncorey@microsoft.com>1 parent bcf8e15 commit 43bcec7
6 files changed
Lines changed: 1605 additions & 0 deletions
File tree
- src/mobius
- models
- tasks
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
124 | 125 | | |
125 | 126 | | |
126 | 127 | | |
| 128 | + | |
| 129 | + | |
127 | 130 | | |
128 | 131 | | |
129 | 132 | | |
| |||
258 | 261 | | |
259 | 262 | | |
260 | 263 | | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
261 | 269 | | |
262 | 270 | | |
263 | 271 | | |
| |||
0 commit comments