Update design to reflect latest thinking

[negz]

The team spent a lot of time on Modelplane and its API last week. The most significant change we've aligned on is to dial down the separation of concerns on deploying a model. Specifically we've dropped the model catalog and we're going to expose more knobs to the ML teams authoring a ModelDeployment.

We've made ModelDeployment much more expressive in order to make sure it could deploy a frontier (open weight) model. We've adopted a subset of DRA to express scheduling constraints. We've also decided to scale only at the replica level - i.e. a ModelDeployment can scale the number of ModelReplicas but there's no scaling within one ModelReplica.

[bassam]

I think we should drop the source discriminator and huggingFace: block on ModelDeployment. I don't think they represent how LLMs are actually packaged today and they get in the way of future caching features..

LLM serving has settled into three distinct factorings of (runtime, weights, optional compiled engine):

Pattern 1 — engine fetches weights at startup. Generic engine image (vLLM, SGLang, TGI upstream); engine pulls weights from HuggingFace, S3, or NGC via its native mechanism. The deployment specifies engine.image and engine args; the engine handles the rest. This is the dominant pattern for vLLM/SGLang/TGI workloads.

Pattern 2 — engine image includes weights. NIM is the canonical case. Runtime, optimization metadata, and weights are baked into one OCI image. Pull, run, serve. No separate fetch step.

Pattern 3 — runtime and artifacts stored separately. Generic runtime image plus separately-stored artifacts: weights the platform fetches and stages, compiled TensorRT-LLM .engine files in object storage, weights in an internal registry, or weights pre-staged on a PVC. Runtime mounts the artifacts at a known path and reads from there.

For pattern 1, source is redundant with what the engine arg already says. If the user writes engine.args: ["--model=meta-llama/Llama-3.1-70B-Instruct"], having a parallel source: { huggingFace: { repo: meta-llama/Llama-3.1-70B-Instruct } } field means the user has to keep two fields in sync, and Modelplane has to decide whether to download via storage initializer (and then have the engine read from a mount) or let the engine fetch directly. Either way, one of the two configurations is doing nothing.

For pattern 2, source is meaningless. The image is the source. There's nothing external to point at.

For pattern 3, source could point at the external artifacts and have Modelplane stage them. This works, but it's the case where I believe we want the primitive to be at fleet-level — a future separate ModelCache resource that names artifacts see #66. Putting source on ModelDeployment scopes the staging to a single deployment, which is exactly the wrong scope for the cases where staging matters.

I propose we drop source and the huggingFace: block from ModelDeployment for v0.1. Patterns 1 and 2 work entirely from engine.image and engine.args. Pattern 3 waits for ModelCache in v0.2, which is also the right layer for fleet-aware caching optimizations.

For private repo access (HF tokens, NGC API keys, S3 credentials), engines already accept env vars. Add an env field on engine mirroring standard PodSpec env, with valueFrom: secretKeyRef for pull tokens:

spec:
  engine:
    name: vLLM
    version: "0.8.5"
    image: vllm/vllm-openai:v0.8.5
    imagePullSecrets:
    - name: nvcr-creds
    env:
    - name: HF_TOKEN
      valueFrom:
        secretKeyRef:
          name: hf-token
          key: token
    args:
    - "--model=moonshotai/Kimi-K2-Instruct"
    - "--tensor-parallel-size=8"
    - "--pipeline-parallel-size=2"

The Kimi K2 example becomes simpler — no source: block, no huggingFace.secretRef, just the engine config. Same for Mixtral, Qwen3-Coder, and the disagg example.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates the Modelplane v0.1 design to reflect the latest API direction: removing the model catalog split, making ModelDeployment self-contained and expressive enough for frontier models, and shifting to replica-level fleet scheduling and scaling.

Changes:

• Replaces InferenceEnvironment/ModelPlacement/ClusterModel with a new resource model (InferenceCluster, InferenceClass, ModelReplica, ModelService, ModelEndpoint).
• Defines two-level scheduling (cluster label selection + node/pool CEL matching) and documents fleet scheduling + replica-only autoscaling.
• Updates examples and diagrams to reflect the new deployment and routing model.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dennis-upbound]

Yay!!!