Propose and implement unopinionated ModelDeployment engines

[negz]

Description of your changes

Fixes #52.

Today spec.workers.topology describes a deployment with parallelism axes (tensor, pipeline, data, dataLocal). Each axis derives and injects engine-specific flags - breaking the pass-through property Modelplane relies on everywhere else, and creating two sources of truth (the user writes the same flags in args).

I propose we describe a deployment by its shape instead. spec.engines is an array of inference engines, each a Standalone member or a Leader and one or more Worker members. A member carries its own nodeSelector and engine template; an engine may be stamped out a fixed number of times with copies.

A small model on a single GPU shows the API at its simplest:

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  replicas: 1
  clusterSelector:
    matchLabels:
      modelplane.ai/tier: production
  engines:
  - name: qwen3-8b
    members:
    - role: Standalone
      nodeSelector:
        devices:
        - name: gpu
          count: 1
          selectors:
          - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("16Gi")) >= 0
      template:
        spec:
          containers:
          - name: engine
            image: vllm/vllm-openai:v0.11.0
            args:
            - --model=Qwen/Qwen2.5-7B-Instruct

This PR ports today's functionality onto the new shape. The scheduler co-schedules a replica's engines onto one cluster, each on a pool that satisfies its members' selectors. compose-model-replica composes a workload per engine - a Deployment for a Standalone, a LeaderWorkerSet for a gang - fronted by one Service and HTTPRoute. Modelplane injects no engine flags; the only injection is MODELPLANE_LEADER_ADDRESS, a backend-neutral alias for the gang leader's address.

Prefill/decode disaggregation (spec.serving and PrefillDecode) is designed here but not implemented.

Both paths were validated end to end on EKS L4 GPUs: a Standalone engine on one GPU, and a Leader/Worker gang serving Qwen2.5-7B pipeline-parallel across two single-GPU nodes, with completions flowing through the control-plane gateway. The multi-node example is the validated manifest. The run also filed #139, #140, and #141 for pre-existing issues it surfaced.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

[dennis-upbound]

i like it!

[Copilot]

Pull request overview

Adds a draft design proposal to make ModelDeployment less engine/opinionated by describing deployment shape (worker groups) and serving (unified vs disaggregated), plus an accompanying “generic” example manifest; updates the existing design doc to mark the current ModelDeployment section as superseded by the new proposal.

Changes:

• Added a new design document describing the proposed unopinionated ModelDeployment worker-group + serving model.
• Added a new example manifest for a disaggregated multi-node deployment shape.
• Annotated design/design.md to point readers to the new proposal and note which parts are superseded.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 15 comments.

File	Description
examples/deployment/generic.yaml	New example manifest demonstrating the proposed schema and disaggregated multi-node shape.
design/unopinionated-deployments.md	New draft design doc describing the proposed unopinionated ModelDeployment API shape and scheduling/serving model.
design/design.md	Marks the existing ModelDeployment spec description as superseded by the new proposal doc.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dennis-upbound]

Two things worth pinning while serving is still fluid, both about how far it stretches:

EPD is coming, and serving.disaggregation is 2-role-shaped. The block names exactly prefillGroupName + decodeGroupName, and the role-labeling/routing that hangs off it assumes two phases. But encode-prefill-decode (a third "encode" stage for multimodal — vision/audio encoders on their own GPUs) is already shipping: SGLang has elastic encoder disaggregation (LMSYS, Jan 2026), llm-d-inference-scheduler is extending P/D→EPD (#608), and the routing sidecar already takes an --ec-connector for the encoder→prefiller hop. workers[] is N groups so the shape holds, but the 2-named-group block would need a third role (or a phases list) to express EPD. Cheaper to leave the door open now than to version the block later.

(More #34 than #137, flagging anyway: KV transfer is shifting from direct prefill→decode to tiered KV stores — Mooncake/LMCache, now NIXL backends (Mooncake, LMCache). The connector's a user flag, but a shared store is a stateful component with no home in workers[].)

[negz]

@dennis-upbound Good point, I did read a little about EPD while working on this. The good part is the encode stage should model cleanly under workers as a third group. Maybe we switch from Unified/Disaggregated to an enum that describes how it's disaggregated? So:

spec:
  serving:
    mode: DisaggregatedPD
    pd:
      # ...

spec:
  serving:
    mode: DisaggregatedEPD
    epd:
      # ...

[dennis-upbound]

_place_engine_split splits an engine's members across pools when no single pool fits them all, and the _place_engine comment calls that "silently degrade interconnect." For a multi-node gang I think it's worse than degrade.

A Leader+Worker engine doing TP/PP/EP talks over its pool's fabric — NVLink within a node, InfiniBand within a pool. Put the leader on one pool and the worker on another and that fabric isn't there: cross-pool tensor parallel never forms and the gang just sits NotReady. The split hands back a placement that can't run, not a slower one, with no signal saying why.

Would it be safer to fail closed? If an engine has more than one claiming member and no single pool fits it, reject with InsufficientCapacity instead of splitting. A false reject is recoverable — add capacity, or size the gang to a pool — but a gang split across fabrics hangs silently. The claimless-coordinator ride-along you describe is fine as-is (it stays on a sibling's pool, one gang one fabric); it's specifically splitting members that each claim their own GPUs across pools that I'd guard.

[dennis-upbound]

Looks great!