Skip to content

Add vLLM OCI artifact format support with media type application/vnd.docker.ai.vllm.v1#7

Draft
Copilot wants to merge 3 commits into
mainfrom
copilot/fix-7a65d069-1749-460b-bf2d-4db57344eea9
Draft

Add vLLM OCI artifact format support with media type application/vnd.docker.ai.vllm.v1#7
Copilot wants to merge 3 commits into
mainfrom
copilot/fix-7a65d069-1749-460b-bf2d-4db57344eea9

Conversation

Copy link
Copy Markdown

Copilot AI commented Sep 11, 2025

This PR adds support for vLLM models as a new OCI artifact format alongside the existing GGUF support. The implementation introduces the media type application/vnd.docker.ai.vllm.v1 for vLLM models and enables seamless packaging and distribution of both GGUF and vLLM formats.

Changes Made

New OCI Media Type

  • Added MediaTypeVLLM constant with value application/vnd.docker.ai.vllm.v1
  • Added FormatVLLM constant for vLLM format identification

vLLM Support Infrastructure

  • Created internal/vllm package with complete model creation and handling logic
  • Added FromVLLM() function to the builder package for creating vLLM model artifacts
  • Created Dockerfile.vllm for containerizing vLLM models

Enhanced CLI Tool

The CLI tool now automatically detects model format based on file extensions:

  • .gguf files → GGUF format (existing behavior)
  • .safetensors and .bin files → vLLM format (new)
  • Unknown extensions → defaults to GGUF with warning (backward compatibility)

Updated Interfaces

Extended existing interfaces to support both formats:

  • Added VLLMPaths() method to Model interface
  • Added VLLMPath() method to ModelBundle interface
  • Updated bundle unpacking logic to handle vLLM model files

Examples

Packaging a vLLM model:

model-distribution-tool package ./model.safetensors registry.example.com/models/llama:v1.0

Packaging a GGUF model (unchanged):

model-distribution-tool package ./model.gguf registry.example.com/models/llama:v1.0

The tool will automatically detect the format and use the appropriate media type in the OCI manifest.

Testing

  • Added comprehensive test coverage for vLLM functionality
  • All existing tests continue to pass (no breaking changes)
  • Manual testing verified both formats work correctly
  • Verified OCI manifests contain correct media types

Backward Compatibility

This is a purely additive change that maintains full backward compatibility. Existing GGUF workflows continue to work exactly as before, while new vLLM workflows are now supported alongside them.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits September 11, 2025 19:40
…docker.ai.vllm.v1

Co-authored-by: ericcurtin <1694275+ericcurtin@users.noreply.github.com>
Co-authored-by: ericcurtin <1694275+ericcurtin@users.noreply.github.com>
Copilot AI changed the title [WIP] There is a application/vnd.docker.ai.gguf.v3 OCI artifact format that can be used with llama.cpp in this repo, create and add another OCI artifact format for vLLM. Add vLLM OCI artifact format support with media type application/vnd.docker.ai.vllm.v1 Sep 11, 2025
Copilot AI requested a review from ericcurtin September 11, 2025 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants