[experimental only] Add onnx-genai InferenceMetadata export (--runtime onnx-genai) - #398
Merged
Merged
Conversation
Emit runtime-supported attention capabilities and bound the default serving KV capacity to 4096 tokens, with a configurable --max-length override. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an onnx-genai integration path that emits inference_metadata.yaml for Mobius exports, alongside a new CLI --runtime onnx-genai mode to generate that sidecar after saving the ONNX package.
Changes:
- Introduces
mobius.integrations.onnx_genaiwith helpers to generate and writeinference_metadata.yaml. - Adds CLI support for
mobius build --runtime onnx-genaiplus--max-lengthto control the emitted serving KV capacity. - Updates
ModelPackage.save()docs to reference the new onnx-genai metadata generation path.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/mobius/integrations/onnx_genai/inference_metadata.py | New generator/writer for onnx-genai inference_metadata.yaml. |
| src/mobius/integrations/onnx_genai/inference_metadata_test.py | Unit tests covering metadata generation, YAML formatting, and validation. |
| src/mobius/integrations/onnx_genai/init.py | Public exports for the onnx-genai integration API. |
| src/mobius/_model_package.py | Docstring update referencing onnx-genai metadata export option. |
| src/mobius/main.py | Adds --runtime onnx-genai flow and --max-length validation/plumbing. |
Comment on lines
+69
to
+73
| capabilities = ["grouped_query_attention" if is_gqa else "multi_head_attention"] | ||
|
|
||
| attention: dict[str, Any] = { | ||
| "type": "group_query_attention" if is_gqa else "multi_head_attention", | ||
| "num_kv_heads": num_kv_heads, |
Comment on lines
+33
to
+36
| "attention": { | ||
| "type": "group_query_attention", | ||
| "num_kv_heads": 2, | ||
| "num_attention_heads": 14, |
| " - grouped_query_attention\n" | ||
| "model:\n" | ||
| " attention:\n" | ||
| " type: group_query_attention\n" |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
justinchuby
marked this pull request as draft
July 14, 2026 17:49
justinchuby
marked this pull request as ready for review
July 16, 2026 15:16
justinchuby
changed the title
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Mobius integration that emits onnx-genai's own
inference_metadata.yaml(the InferenceMetadata schema) instead of ORT-GenAI'sgenai_config.json, so onnx-genai models carry the config that runtime needs.--runtime onnx-genaibuild target; emittersrc/mobius/integrations/onnx_genai/inference_metadata.py.model.attention(type/heads/head_dim),model.max_sequence_length,kv_cache.native_dtype, andrequired_capabilitiesusing the runtime's supported strings (grouped_query_attention/multi_head_attention).min(4096, model limit)(avoids exceeding WebGPU's 256 MiB buffer limit for large context windows); optional--max-lengthoverride.inference_metadata.yamlalone (no genai_config.json) and produces coherent output.lintrunner clean; emitter pytest 8 passed.