Add Foundry Local deployment skill - #243
Merged
Merged
Conversation
Document the workflow for deploying mobius-exported ONNX models to Microsoft Foundry Local: - Custom model registration via the cache directory - inference_model.json format with common prompt templates (ChatML, Llama, Gemma) - CLI (foundry model run) and OpenAI-compatible API access - Python SDK usage and discovery limitations - Known limitations: bundled GenAI version, no CUDA in pip SDK, multimodal requires GenAI 0.14+, model_type whitelist workaround - Complete end-to-end example from export to local inference Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new agent skill documenting how to take a mobius ORT GenAI export and register/run it in Microsoft Foundry Local. This fits the repo’s skills library by covering downstream deployment of exported ONNX packages.
Changes:
- Adds a new
.agentsskill for Foundry Local deployment and troubleshooting. - Documents cache-based custom model registration and
inference_model.json. - Includes CLI/API usage, limitations, and an end-to-end example workflow.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
titaiwangms
reviewed
May 4, 2026
titaiwangms
left a comment
titaiwangms
left a comment
Contributor
There was a problem hiding this comment.
Nice skill doc! A few suggestions from our Pixtral/Ministral-3 VLM work:
1. Missing processor_config.json in copy step
Pixtral/Ministral-3 VLM exports use processor_config.json (not image_processor.json) for the vision preprocessing pipeline. Without copying this file, VLM inference will fail — the PixtralImageSizes and other preprocessing transforms won't be loaded. Suggest adding:
cp output/processor_config.json "${CACHE_DIR}/models/Custom/${MODEL_NAME}/" 2>/dev/null2. Missing single-component model copy
Text-only LLMs export as model.onnx + model.onnx.data at the output root (not in a decoder/ subdirectory). The copy commands only handle multi-component subdirs. Suggest adding:
cp output/model.onnx* "${CACHE_DIR}/models/Custom/${MODEL_NAME}/" 2>/dev/null3. model_type path is nested
The workaround says to set model_type: "decoder" but the actual key path in genai_config.json is model.type (nested under the model object), not a top-level key.
4. Mistral3 prompt template (nice-to-have)
The skill includes ChatML, Llama, and Gemma templates but not Mistral3's [INST]/[/INST] format. Since Ministral-3 is now supported via mobius, this would be a helpful addition.
Address all 4 review findings: 1. Fix prerequisites to describe both single-model (root-level model.onnx) and multi-model (decoder/, embedding/, etc.) layouts. 2. Fix copy commands to handle single-model exports (cp model.onnx* and *.safetensors from root) plus a simpler cp -r output/* alternative. 3. Add processor_config.json to the copy list alongside image_processor.json — needed for Pixtral/Mistral3 vision exports. 4. Fix model_type workaround: correct JSON path is model.type (not top-level model_type). Added JSON snippet showing the correct structure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Member
Author
|
@titaiwangms Done |
titaiwangms
approved these changes
May 4, 2026
… Local skill Add prominent WARNING in prerequisites about foundry-local-sdk overriding custom ORT/GenAI builds, with the force-reinstall workaround (symlink replacement works because Foundry's native core links to Python-installed .so files). Add 'Building ORT and GenAI from source' section linking to issue #245 for the full build tutorial. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
…ocal New skill at .agents/skills/building-ort-genai/SKILL.md covering: - CUDA toolkit (12.8+/13.0) and cuDNN 9.x setup - ORT build from source with all CUDA flags (flash attention, architecture selection, cuDNN paths) - ORT install layout creation for GenAI linking - GenAI build from source linked to custom ORT - Build verification commands - 6 common issues: cuDNN mismatch, LD_LIBRARY_PATH, pip overrides, ABI mismatch, test target failures, flash attention Updated foundry-local skill to cross-reference the new skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Add specific download URLs for CUDA 12.8 and cuDNN 9.8, concrete install paths ($HOME/cuda12.8, $HOME/cudnn9.8), and the pip-based cuDNN alternative. Restructured to 7 clear steps with explicit commands throughout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Update all CUDA references from 12.8 to 13.0: download URL, install path ($HOME/cuda13.0), PATH and LD_LIBRARY_PATH examples, and build flags. cuDNN 9.8 (cuda12 archive) remains — it's compatible with both CUDA 12.x and 13.x. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Change nvidia-cudnn-cu12 pip package to nvidia-cudnn-cu13 and update cuDNN archive filename to cuda13 variant to match CUDA 13.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Member
Author
|
Will improve build instruction if needed |
This was referenced May 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Document how to deploy mobius-exported ONNX models to Microsoft Foundry Local.
What this adds
New skill at
.agents/skills/foundry-local/SKILL.mdcovering:foundry model run), OpenAI-compatible API, Python SDKmobius buildto local inferenceReference: https://techcommunity.microsoft.com/blog/educatordeveloperblog/deploying-custom-models-with-microsoft-olive-and-foundry-local/4489002
Replaces #242 (closed due to wrong branch name).