Disaggregate prefill and decode: prefill block, role-aware scheduler, and InferencePool routing

[dennis-upbound]

Fixes #34.

LLM inference has two phases with opposite hardware profiles: prefill is compute-bound and sets TTFT, decode is memory-bandwidth-bound and sets ITL. Run on one pod set they contend, and neither can be tuned independently. Modelplane had no way to split them.

This serves the two phases as separate, co-located pod sets with the KV cache transferred over NIXL, and sequences each request prefill→decode through a GAIE InferencePool and an endpoint-picker. It implements the design from #116.

A deployment opts in by declaring a prefill block alongside workers (now the decode role), plus a routing block that supplies the endpoint-picker. For a disaggregated deployment both worker counts and routing must be explicit, enforced by CEL; a deployment with no prefill block is unified serving and routes exactly as before.

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
spec:
  model: { ... }
  workers:                 # decode role
    count: 2
    topology: { tensor: 1 }
    template: { spec: { containers: [{ name: engine, image: ... }] } }
  prefill:                 # new: a second, independently-sized role
    workers:
      count: 1
      topology: { tensor: 1 }
      template: { spec: { containers: [{ name: engine, image: ... }] } }
    nodeSelector: { ... }
  routing:                 # new: the endpoint-picker
    template:
      spec:
        containers:
        - name: epp        # image/args optional; defaults to the pinned llm-d EPP

The scheduler (compose-model-deployment) treats a disaggregated replica as the existing decode placement plus one optional prefill placement, co-located on one InferenceCluster. It chooses the (decode_pool, prefill_pool) pair jointly against a single capacity ledger rather than greedily per role, charges both pools, and re-places the replica if either pool drifts.

The llm-d backend (compose-model-replica) emits a decode pod set (kv_consumer) and a prefill pod set (kv_producer), each role-labeled with llm-d.ai/role and pinned to its pool with its own ResourceClaimTemplate, both carrying VLLM_NIXL_SIDE_CHANNEL_HOST via the downward API so NixlConnector can establish the transfer. It injects the pd-sidecar on decode pods (vLLM moves to 8001, the sidecar takes 8000), emits the InferencePool and the EPP stack (Deployment, Service, ConfigMap, RBAC) built from routing.template, and points the disaggregated HTTPRoute at the pool instead of the Service.

ServingStack installs Envoy AI Gateway (v0.7.0) and the GAIE CRDs, because core Envoy Gateway can't serve an InferencePool; the Phase 1 spike (design/disaggregation-routing-spike.md) established this and the follow-up spike confirmed the mechanism against upstream source.

The unified path is the zero-prefill case of the same code, unchanged throughout.

Validated on real GKE L4 GPUs, which caught four bugs unit tests couldn't — the expected manifests encoded the same wrong values — all fixed here:

• The pinned EPP and pd-sidecar image refs didn't exist (403 from ghcr.io). Corrected to the published public images ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0 and ghcr.io/llm-d/llm-d-routing-sidecar:v0.8.0; the EPP image now pulls and runs.
• The EndpointPickerConfig used apiVersion: llm-d.ai/v1alpha1, which the EPP binary doesn't register (crash-loop). Corrected to inference.networking.x-k8s.io/v1alpha1; the EPP now parses its config, runs, and reconciles the InferencePool (tracking both the decode and prefill endpoints).
• The disaggregated engines never enabled vLLM's NixlConnector — the side-channel env and sidecar were emitted but no --kv-transfer-config, so no prefill→decode KV handoff could occur. Both roles now pass --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' (NixlConnector doesn't distinguish kv_role; the routing sidecar drives direction). Confirmed live: both engines come up with the connector configured and log NIXL is available.
• The pd-sidecar served HTTPS by default (--secure-proxy defaults true), so the HTTP readiness probe and the HTTP gateway path were rejected and the decode pod never became Ready. Now passes --secure-proxy=false (the Modelplane serving path is HTTP throughout).

With these, both vLLM roles run on L4 with NixlConnector active and the model serves real completions — e.g. a chat request returned Hello! How can I assist you today?. The EPP is healthy and tracks both endpoints.

Not yet confirmed: a single request driven end to end through the gateway (HTTPRoute → InferencePool → EPP → decode, sidecar pulling KV from prefill). On the test cluster the InferencePool backendRef didn't program, but the substrate had been disturbed during debugging, so this is inconclusive rather than a confirmed limitation and needs a clean re-run. Envoy AI Gateway is pinned at v0.7.0 pending its v1.0.0 GA (~end of June).

Follow-ups, out of scope and pre-existing: (1) a product-provisioned GKE cluster brings GPU pools up on the device-plugin path while workloads claim GPUs only via the gpu.nvidia.com DRA driver, so GKE-side DRA provisioning (driver-root, device-plugin) needs validating — affects unified serving too; (2) the LWS bootstrap runs ray start unconditionally, but stock vllm/vllm-openai ships no ray and single-node replicas don't need it.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

[negz]

Superseded by #142