Add NeMo .nemo integration + FastConformer-RNNT streaming ASR model - #359
Conversation
Introduce mobius.integrations.nemo, mirroring the GGUF integration, to import NVIDIA NeMo .nemo archives: - NeMoArchive reads the tar bundle without depending on nemo_toolkit: parses model_config.yaml, loads the model_weights.ckpt state_dict (weights_only), extracts SentencePiece tokenizer artifacts, and resolves nemo:<file> URIs. Accepts a local path or HF Hub reference. - nemo_to_config maps the FastConformer-RNNT (EncDecRNNTBPEModel) config to an ArchitectureConfig, adding a focused group of FastConformer/RNN-T fields to ArchitectureConfig. - build_from_nemo wires reader → config → registry → build_from_module → weight loading (lazy imports; the registry model class lands in a follow-up commit). Verified against the real nemotron-speech-streaming-en-0.6b.nemo: 653 parameters load and config dimensions map correctly. 17 offline unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Implement the FastConformer-RNNT transducer architecture used by nvidia/nemotron-speech-streaming-en-0.6b, building on the new .nemo integration. The model is emitted as three ONNX sub-models (encoder / prediction decoder / joint) wired through a new RNNTTask. What's added: - models/nemo_rnnt.py: full EncDecRNNTModel — causal dw_striding conv subsampling (8x), relative-position multi-head attention with rel_shift, Conformer layers (FF/attn/conv/FF + norms), 2-layer LSTM prediction net with PyTorch->ONNX gate reordering, and a log-softmaxed joint network. - tasks/_rnnt.py: 3-model RNNTTask (encoder/decoder/joint) with documented offline full-context, batch=1 runtime contract. - integrations/nemo: config mapping now validates encoder architecture assumptions (dw_striding/rel_pos/layer_norm/no-xscaling) and the builder rejects non-fp32 export, failing loudly on unsupported variants. - Registry, models/tasks exports, and ArchitectureConfig fields wired up. Tests: - L1 unit tests (graph build + I/O shapes) via random-weight fill. - L4 parity vs a committed NeMo golden: encoder ~5e-7, decoder ~5e-7, joint ~2.7e-5. - L5 greedy decode smoke test plus an incremental-vs-one-shot decoder state-consistency check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Performance Comparison
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
There was a problem hiding this comment.
Pull request overview
Adds first-class support for importing NVIDIA NeMo .nemo archives into mobius and introduces a FastConformer-RNNT (RNN-T) speech recognition model/task that exports as three ONNX sub-models (encoder / prediction decoder / joint).
Changes:
- Added a NeMo
.nemointegration layer (archive reader, config→ArchitectureConfig mapping, andbuild_from_nemobuild pipeline). - Added FastConformer-RNNT model implementation and a new
RNNTTaskthat emits encoder/decoder/joint ONNX graphs. - Added unit + integration tests for graph validity and parity/greedy decoding against a NeMo golden reference.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/nemo_rnnt_integration_test.py | Integration parity + greedy decode smoke tests using real .nemo + golden NPZ. |
| tests/build_graph_test.py | Registers fastconformer_rnnt as a specialized model type for build coverage. |
| src/mobius/tasks/_rnnt.py | New RNNTTask exporting encoder/decoder/joint graphs and defining I/O contracts. |
| src/mobius/tasks/init.py | Exposes RNNTTask and registers the "fastconformer-rnnt" task name. |
| src/mobius/models/nemo_rnnt.py | Implements FastConformer encoder + RNN-T prediction/joint modules + NeMo weight mapping. |
| src/mobius/models/nemo_rnnt_test.py | Unit tests that build/run the three graphs with random weights to validate shapes/structure. |
| src/mobius/models/init.py | Exports EncDecRNNTModel. |
| src/mobius/integrations/nemo/_reader.py | Implements NeMoArchive for config/weights/tokenizer extraction from .nemo. |
| src/mobius/integrations/nemo/_reader_test.py | Offline tests for NeMoArchive using a synthetic .nemo tar. |
| src/mobius/integrations/nemo/_config_mapping.py | Maps NeMo model_config.yaml into ArchitectureConfig with validation guards. |
| src/mobius/integrations/nemo/_config_mapping_test.py | Tests for config mapping + unsupported-config rejection. |
| src/mobius/integrations/nemo/_builder.py | Adds build_from_nemo pipeline: read → map → registry resolve → build → apply weights. |
| src/mobius/integrations/nemo/init.py | Public entrypoint export for build_from_nemo. |
| src/mobius/_registry.py | Registers fastconformer_rnnt → EncDecRNNTModel with task "fastconformer-rnnt". |
| src/mobius/_configs/_base.py | Adds FastConformer-RNNT-related fields to ArchitectureConfig. |
Address review gaps on the FastConformer-RNNT PR: - Expose the `.nemo` build path like GGUF: export `build_from_nemo` from the top-level `mobius` package and add a `mobius build-nemo` CLI subcommand (local path or HF repo, dtype/EP/output options). - Add `revision=` support threaded through `build_from_nemo` -> `NeMoArchive` -> `_resolve_nemo_path` so HF downloads can be pinned. - Pin the integration test to the model's HF commit SHA and store self-describing metadata (model id, revision, NeMo version, dtype, seed, token/SOS/blank ids) in the committed golden npz. - Commit a reproducible golden generator (scripts/generate_nemo_rnnt_golden.py). - Import shared components from the public `mobius.components` API and drop a dead TYPE_CHECKING block in models/nemo_rnnt.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Follow-up: review-gap fixes (commit e7c35d4)
Quality-checklist scope (explicit N/A waivers)These standard items don't apply to this model and are intentionally skipped:
|
…-RNNT Two follow-ups requested after the initial PR: fp16/bf16 export: - Cast dtype-sensitive scalar constants (attention scale, mask fill, macaron factor) to the compute dtype via CastLike, and cast the float32 Sin/Cos positional encoding to the compute dtype before projection. - Builder no longer rejects non-fp32; f16/bf16 now build end-to-end. Validated the f16 encoder against the f32 golden on CUDA (maxdiff ~1e-3). Ragged-batch support: - Encoder gains a `length` (B,) input and an `encoder_length` (B,) output. - Subsampled lengths follow NeMo's calc_length (n -> floor(n/2)+1 x3); a per-frame validity mask gates attention (queries never attend to padded keys) and zeroes padded output frames. - length=full reduces exactly to the previous full-context path, so NeMo golden parity is preserved. Tests: half-precision build/type-check (f16/bf16), padding-mask consistency (a sample's valid region is identical alone vs padded in a batch), and a CUDA-gated f16 encoder parity test. Updated golden/greedy tests to pass length. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Add a NeMo cache-aware streaming export for the FastConformer-RNNT
encoder, alongside the existing offline full-context encoder. The new
"encoder_streaming" model in the RNNT package consumes one feature chunk
plus NeMo's per-layer running state (cache_last_channel, cache_last_time,
cache_last_channel_len) and returns the chunk encoding together with the
updated caches, enabling chunk-by-chunk inference.
What:
- Streaming paths threaded through the encoder components via optional
cache kwargs on forward() (so all calls still go through nn.Module
__call__ for correct initializer naming):
- _Conv1d: causal conv using a left-context cache instead of zero pad;
returns the updated cache (last k-1 frames of concat(cache, x)).
- RelPositionMultiHeadAttention: non-square q/kv attention (queries from
the current chunk, keys/values from concat(cache, chunk)) with a
relative-position embedding spanning the full window.
- ConformerConvolution / ConformerLayer: cache_time / cache_channel
threading; the cached attention state is the normed attention input.
- FastConformerEncoder._forward_streaming: drop_extra_pre_encoded,
cache-aware mask with chunk-aligned offset, per-layer cache stacking,
and cache_last_channel_len growth capped at cache_size.
- RNNTTask emits "encoder_streaming" (5 cache I/O tensors). The streaming
graph is built from a deep copy of the module so its parameters realize
as independent initializers (same names) in a separate graph.
- Config: derive fastconformer_streaming_cache_size (= att_context left)
and fastconformer_streaming_drop_extra from the NeMo encoder config.
Validation:
- New streaming reference generator
(scripts/generate_nemo_rnnt_streaming_golden.py) + compact golden.
- L1 unit test (I/O shapes + cache growth) and L4 chunk-chained parity
test vs NeMo: enc_out matches to ~1e-6 over two chunks; cache lengths
exact. Full non-integration suite (2908) and lint pass.
Streaming targets a single stream of equal-length chunks (batch=1 /
homogeneous, no intra-chunk padding), matching NeMo's cache-aware design;
ragged batches should use the offline "encoder".
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Cache-aware streaming encoder added (commit c07dda4)The third follow-up is done. The RNNT package now also exports an Contract: inputs Implementation: streaming threaded through the components via optional cache Validation: new reproducible streaming golden generator + compact golden. All three documented follow-ups (fp16/bf16, ragged-batch length masking, |
Add examples/nemotron_fastconformer_rnnt.py demonstrating both inference modes for the NeMo FastConformer-CacheAware-RNNT model: * Offline (from file): full-context encoder + RNN-T greedy decode for a complete transcript. * Real-time (streaming): chunked, cache-aware encoder_streaming model carrying attention/conv caches across chunks, printing partial text as each chunk is decoded; the same loop drives microphone input. The example is self-contained: the log-mel frontend (matching NeMo's AudioToMelSpectrogramPreprocessor, normalize="NA") and the SentencePiece BPE tokenizer are reconstructed from the .nemo archive, so nemo_toolkit is not a runtime dependency. Validated end-to-end on a LibriSpeech sample: offline transcript is exact; streaming converges to the same text. The mel frontend matches NeMo's preprocessor on interior frames (mean abs diff ~2e-4). Streaming chunk length defaults to the model's native chunk (att_context [70,13] -> 1.12 s) and is snapped to an 8-frame subsampling multiple. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Address rubber-duck and code-review feedback on the FastConformer-RNNT example: * RnntGreedyDecoder.reset now preserves the prediction-network LSTM state produced by the SOS step (previously discarded), so the first emitted token is conditioned on the post-SOS state, matching NeMo add_sos=True. * Hold the tokenizer's TemporaryDirectory on the pipeline instance instead of leaking a mkdtemp directory on every run. * Microphone streaming now starts with empty left-context (consistent with the first file-streaming chunk) and carries the overlap from the running audio buffer, robust to short blocks. * Reject non-positive --chunk-seconds. Re-validated: offline transcript still exact and streaming still converges on a LibriSpeech sample; 31 nemo unit tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
* Add fastconformer_rnnt to _COVERAGE_SKIP in model_coverage_test.py: it is a NeMo .nemo RNN-T ASR model loaded from a .nemo archive (no standard HF config / test_model_id), and is covered by tests/nemo_rnnt_integration_test.py. Fixes the four failing TestL1L3GraphBuildCoverage / TestL2ConfigValidation cases. * Escape the regex metacharacter in the pytest.raises match pattern in _reader_test.py (RUF043), which CI lint (check mode) treats as a failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Adapt the FastConformer-RNNT ONNX export so the ONNX Runtime GenAI
`nemotron_speech` C++ pipeline can consume it directly, and add a
deployment-bundle generator.
Graph layout (tasks/_rnnt.py): the `encoder_streaming` and `joint`
graphs now follow the GenAI contract at the I/O boundary via transposes,
while the model `forward()` methods keep their native NeMo-parity layout:
- encoder_streaming: time-major audio `(B, T, mel)` and output
`(B, T, d)`; batch-first caches `(B, L, 70, d)` / `(B, L, d, 8)`.
- joint: single-frame time-major inputs `(B, 1, d)` / `(B, 1, d_pred)`.
- decoder: already compatible (start token = blank_id 1024 maps to the
zero SOS embedding row); unchanged.
The offline `encoder` graph intentionally keeps the native feature-major
layout (it is not part of the GenAI streaming pipeline). The unit test,
integration test and example decode loops are updated to the new layouts.
Bundle (integrations/nemo/_genai_config.py): `write_genai_bundle` writes
flat encoder/decoder/joint ONNX plus genai_config.json,
audio_processor_config.json, an HF Unigram tokenizer.json derived from the
.nemo SentencePiece model (Metaspace decoder so ▁ marks decode to spaces),
and an optional Silero VAD download. Wired into `mobius build-nemo --genai`.
The GenAI runtime only supports float32 encoder I/O, so the generator
asserts the package was built in float32.
Validated: integration parity/streaming/greedy/fp16 + bundle load via
onnxruntime_genai; example transcribes the LibriSpeech sample exactly
(offline) and converges (streaming); full fast unit suite (2911) + lint.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Make the FastConformer-RNNT encoder and joint graphs natively time-major / batch-first — the layout already used internally by the model and required by the ORT GenAI nemotron_speech pipeline — so the exported graphs carry no layout transposes at all. Previously the offline encoder emitted feature-major (B, d, T) while the streaming encoder adapted to the GenAI time-major layout via boundary transposes (commit 966dadf), leaving two divergent encoder layouts and redundant transposes. The model's ConvSubsampling already transposed the feature-major input to time-major on entry and the encoder transposed back on exit purely to match NeMo's I/O convention; the joint likewise transposed its inputs internally. Adopting time-major as the native layout removes every one of those transposes and unifies the offline and streaming encoders on a single layout. Changes: - models/nemo_rnnt.py: ConvSubsampling consumes (B, T, feat_in) directly; FastConformerEncoder offline + streaming return time-major (B, T', d) with batch-first caches (B, L, ...); RNNTJoint projects time-major inputs directly. Decoder stays feature-major (B, d_pred, U) per the C++ contract. - tasks/_rnnt.py: declare native time-major audio/output + batch-first caches in encoder/encoder_streaming/joint builders; drop all boundary transposes; update module docstring (encoder now unified time-major). - tests + example: feed/compare in the native time-major layout; the runtime greedy loop keeps the single decoder-frame transpose for the joint. - _genai_config.py: ruff RUF046/TRY300 cleanups. Validated: nemo unit tests, integration parity/streaming/greedy/fp16/genai bundle (5 pass), example offline (exact) + streaming (converges) on LibriSpeech, lint clean, full fast suite green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Resolve actionable review comments on PR #359: - models/nemo_rnnt.py: update the module docstring and ConvSubsampling docstring to the native time-major layout and NeMo's per-stage causal length rule (the previous text described the old feature-major layout and an inaccurate ceil(T/8) length). - _reader.py: expand `~` in `_resolve_nemo_path` so `build_from_nemo('~/m.nemo')` resolves the local file; stream `model_weights.ckpt` straight from the tar member (seekable fast path, buffered fallback) to avoid buffering the multi-GB checkpoint in memory. - __main__.py: add `build-nemo --revision` to pin HF Hub downloads and thread it through `build_from_nemo` and the GenAI `NeMoArchive`; correct the `--dtype` help (f16/bf16 are supported; only the GenAI bundle is f32-only). - _config_mapping.py: drop the unused `logging` import / `logger`. - tests: use `ir.serde.serialize_model` instead of `ir.to_proto`; replace an empty `except ImportError: pass` with `pytest.skip`; restore the sorted order of `_SPECIALIZED_TEST_MODEL_TYPES`. Note: the GenAI bundle's `log_eps` (2**-24) and `audio_processor` `log_zero_guard_value` (1e-10) intentionally differ — both match the official olive nemotron_speech recipe exactly. Validated: nemo unit + coverage tests, integration parity + genai bundle (real .nemo archive, streamed state_dict), lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Replace the manual matrix_ac MatMul + Softmax + context MatMul in RelPositionMultiHeadAttention with the opset-23 fused `op.Attention` op, so ORT can dispatch the scaled-dot-product attention to its memory-efficient / fused kernels. NeMo's Transformer-XL relative-position attention has two score terms: matrix_ac = (q + pos_bias_u) @ k^T and the data-dependent matrix_bd = rel_shift((q + pos_bias_v) @ p^T). The content term matrix_ac is evaluated by op.Attention itself by folding pos_bias_u into the query and letting the op apply the 1/sqrt(d_k) scale; matrix_bd (pre-scaled to the same domain) plus the boolean keep-mask (converted to a -INF_VAL additive bias) are combined into the single additive attention bias the op accepts. Net effect is identical scores with fewer, fused graph nodes. Notes on the two questions this addresses: - GQA does not apply: this is standard MHA with equal Q/KV heads, and the relative-position bias is not expressible by the fused GroupQueryAttention op (which assumes rotary, not Transformer-XL rel_pos). - The existing components/_audio.py ConformerEncoder/ConformerAttention cannot be reused as-is: they implement a T5-relative-bias conformer with symmetric-pad subsampling (Phi4MM family), whereas NeMo FastConformer uses Transformer-XL rel_pos with causal subsampling. This change adopts their op.Attention pattern without taking the incompatible mechanism. Validated: nemo unit tests; integration offline parity (atol 1e-4), fp16 parity (CUDA), streaming parity, greedy decode; example offline exact + streaming converges on LibriSpeech; lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Replace the manual `x * sigmoid(x)` SiLU in the conformer feed-forward and convolution modules with the native opset-24 `op.Swish` op (alpha=1). The model graphs already target OPSET_VERSION=24, and ORT 1.27 implements Swish for float32 (CPU/CUDA), float16 (CPU/CUDA) and bfloat16 (CUDA) — covering every dtype/EP path this model executes — so this emits one fused node instead of a Mul+Sigmoid pair with identical numerics. The conv module's GLU gate stays as `Mul(a, Sigmoid(b))`: ONNX has no GLU/Glu operator (the two operands are different split halves, not a self-gate). Validated: nemo unit tests; integration offline parity (atol 1e-4), fp16 parity (CUDA), streaming parity, greedy decode; example offline exact on LibriSpeech; lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
The codebase emits opset 24 graphs (OPSET_VERSION=24 in src/mobius/_constants.py), but two docs implied opset 23: - .github/copilot-instructions.md said "Use ONNX opset 23 op.Attention". op.Attention was introduced in opset 23 but our graphs target opset 24; clarify both facts. - docs/design/gguf-support-proposal.md said "Our codebase uses opset 23"; corrected to opset 24. Other "opset 23" mentions in the skills correctly refer to op-introduction versions (Attention/RMSNormalization) or the opset-24->23 lowering fallback for older ORT CUDA EPs, and are left unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Review — multi-model team review (readability, correctness, adversarial, spec-adherence, integration)Overall this is a high-quality PR. The FastConformer-RNNT math was independently verified against the NeMo reference source (rel-pos decomposition via fused Major1. Streaming cache length grows by physical chunk frames ( new_len = op.Min(op.Add(cache_last_channel_len, op.Squeeze(tq)), ...)
2. 3. 4. No Minor
Question
Nits
Praise
🤖 Synthesized from a 5-model review team (Claude readability + Opus deep/spec + GPT correctness & adversarial + Gemini integration). Spec claims grounded against NeMo source; the two top Majors verified directly against the diff. |
Cache-aware multi-context NeMo archives may store pre_encode_cache_size as a list (e.g. [4, 9]). nemo_to_config() called int() on it directly, which raised TypeError on those archives. Mirror the genai-config path (and the reference olive recipe) by taking the last entry before int(). Add unit tests for list-valued pre_encode_cache_size and att_context_size. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
After migrating SiLU to the native opset-24 op.Swish, the _swish helper was just a one-line pass-through (return op.Swish(x)). Inline it at the two call sites in the macaron FFN and conv module and remove the wrapper. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Act on the high-value findings from the multi-model PR #359 review: Correctness: - Streaming cache length now grows by the chunk's *valid* frame count (length_sub), not the physical post-drop T_out. A short/final chunk previously over-incremented cache_last_channel_len, letting the next step's mask treat padded cache frames as valid keys. Identical for full chunks (the validated path). Robustness / fail-loud: - _validate_encoder now rejects use_bias=True (the stack hardcodes bias=False; non-strict weight loading would otherwise silently drop every bias) and att_context_style != "chunked_limited" (the only mask rule implemented). - nemo_to_config rejects an unlimited right context (att_context[1] == -1), which would make chunk_size 0 → div-by-zero in the streaming mask. - Pin the Silero VAD download to a fixed revision so include_vad bundles are reproducible. Design: - Store the resolved registry model_type on the native ArchitectureConfig .model_type field instead of monkey-patching config._nemo_model_type, which dataclasses.replace() silently dropped (forcing a re-stash). Nits: - Fix spurious parens around the "fastconformer_rnnt" registry value. - Document why all four RNN-T sub-models map to the "encoder" role. Add unit tests for use_bias, att_context_style, and unlimited-right-context guards. All nemo unit tests (39) and integration tests (5, real model) pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
|
Thanks for the exceptionally thorough multi-model review — the spec-grounding against the NeMo source was especially valuable. Addressed in FixedMajor 1 — streaming cache length ✅ Now grows by the chunk's valid frame count ( Major 2 — Major 3 — Major 4 — Minor — list-valued Minor — unpinned VAD download ✅ Pinned Nits ✅ Fixed the spurious Added 4 unit tests (use_bias / att_context_style / unlimited-right / list configs). All 39 nemo unit + 5 integration tests (real model) pass, lint clean. Respectfully pushing backMinor — genai_config log-eps "inconsistency". Question — fully-masked query row. Confirmed benign and no batched-streaming path can escape it: offline padded rows are re-zeroed via Deferred (doc-only, low-risk)Dead-field round-trip comments, untrusted-archive resource caps, the unused |
The example runs ONNX via mobius._testing.ort_inference.OnnxModelSession, which imports onnxruntime and onnxruntime-easy. These are not pulled in by the mobius-ai core package, so the documented `pip install mobius-ai librosa soundfile sentencepiece` line failed at import with ModuleNotFoundError: onnxruntime_easy. Add the missing packages and note that torch is a core dependency. Also document that only Linux/x86-64 (CPU/CUDA) is verified; the CPU path should work elsewhere but is untested. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
onnxruntime-easy supports device="webgpu"; expose it as a --device choice so it can be tried where an onnxruntime WebGPU EP build is available. Mark it EXPERIMENTAL/unverified: the encoder emits opset-24 Swish + Attention and the decoder uses LSTM, which may lack WebGPU kernels (CPU fallback or errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Drop the dedicated `mobius build-nemo` subcommand (~130 lines: handler + 8-arg subparser) in favor of auto-detecting `.nemo` inputs in the existing `build` command, mirroring how diffusers pipelines are auto-detected. When `--model` ends with `.nemo` (local file or `owner/repo:model.nemo` HF ref), `build` routes to `build_from_nemo` and reuses the standard `--dtype`/`--ep`/ `--external-data` args and `_save_package` save logic. The GenAI `nemotron_speech` bundle path (formerly `--genai`/`--chunk-seconds`/ `--no-vad`) is not part of the core CLI; it remains available via the Python API (`write_genai_bundle`) and the example script. Add test_build_dot_nemo_model_routes_to_nemo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Summary
Adds support for
nvidia/nemotron-speech-streaming-en-0.6b(a FastConformer-CacheAware-RNNT streaming ASR model distributed as a NeMo.nemoarchive) in two layers:New
.nemointegration (src/mobius/integrations/nemo/) — analogous to the existing GGUF integration. An archive reader (NeMoArchive), a config mapping (model_config.yaml→ArchitectureConfig), and abuild_from_nemopipeline. Resolves local paths and HuggingFace Hub references.FastConformer-RNNT model (
models/nemo_rnnt.py+tasks/_rnnt.py) — emitted as three ONNX sub-models (encoder / prediction decoder / joint) via a newRNNTTask.Architecture
dw_stridingconv subsampling → 24 Conformer layers (relative-position MHA withrel_shift, FF/attn/conv/FF macaron structure, layer-norm conv module). Positional encodings built dynamically in-graph.add_sos=True).Validation
Parity validated against
nemo_toolkit2.7.3 reference.Contract / limitations
dw_striding/rel_pos/layer_norm,xscaling) are rejected with a clear error.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com