diff --git a/content/en/docs/next/networking/architecture.md b/content/en/docs/next/networking/architecture.md index e9d8d239..5839811a 100644 --- a/content/en/docs/next/networking/architecture.md +++ b/content/en/docs/next/networking/architecture.md @@ -433,7 +433,7 @@ cilium: enabled: true ``` -See [Enabling Hubble](https://docs.cilium.io/en/stable/observability/hubble/) for full configuration details. +See [Enabling Hubble for network observability](/docs/next/networking/hubble/) for the metrics list this needs, the Grafana dashboards Cozystack ships for it, and troubleshooting. The upstream [Cilium Hubble documentation](https://docs.cilium.io/en/stable/observability/hubble/) covers the remaining knobs. ## Traffic Flow Summary diff --git a/content/en/docs/next/networking/hubble.md b/content/en/docs/next/networking/hubble.md new file mode 100644 index 00000000..4f2f7cad --- /dev/null +++ b/content/en/docs/next/networking/hubble.md @@ -0,0 +1,96 @@ +--- +title: "Enabling Hubble for Network Observability" +linkTitle: "Hubble" +description: "Turn on Cilium's Hubble observability stack, and read the flow, DNS and L7 metrics through the platform Grafana dashboards Cozystack ships for it." +weight: 50 +--- + +Hubble is the network and security observability layer built on top of Cilium. It gives visibility into the communication and behaviour of services in the cluster — flow logs, DNS queries, and L7 request metrics. + +Hubble is **disabled by default** in Cozystack to keep resource usage down. This page covers turning it on and reading the results. For where Hubble sits in the data plane, see [Networking architecture](/docs/next/networking/architecture/#observability-with-hubble). + +## Prerequisites + +- A Cozystack cluster running Cilium as the CNI (the default). +- The [Monitoring](/docs/next/operations/services/monitoring/) hub deployed, for Grafana access and metric storage. + +## Enable Hubble + +Enable Hubble, Relay and the UI in the Cilium configuration, and turn on the metrics you want exported: + +```yaml +cilium: + hubble: + enabled: true + relay: + enabled: true + ui: + enabled: true + metrics: + enabled: + - dns + - drop + - tcp + - flow + - port-distribution + - icmp + - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction +``` + +The `metrics.enabled` list is what makes the dashboards below work — without it Hubble runs but exports nothing for Grafana to draw. The `httpV2` entry in particular must keep its `labelsContext`, because the L7 HTTP dashboard groups by source and destination workload and cannot do so if those labels are absent. + +### Components + +Enabling Hubble brings up: + +- **Hubble Relay** — aggregates flow data from all Cilium agents. +- **Hubble UI** — web interface for exploring network flows. +- **Hubble Metrics** — Prometheus metrics for network observability. + +## Grafana dashboards + +Cozystack ships four Hubble dashboards, delivered in the `hubble` folder of the platform Grafana: + +| Dashboard | Description | +|-----------|-------------| +| **Overview** | General Hubble metrics including processing statistics | +| **DNS Namespace** | DNS query and response metrics by namespace | +| **L7 HTTP Metrics** | HTTP layer 7 metrics by workload | +| **Network Overview** | Network flow overview by namespace | + +These are infrastructure dashboards, so they are provisioned only for the platform-level Monitoring release — the one in `tenant-root` or `cozy-monitoring`. A tenant's own Grafana does not receive them; tenants see their own application dashboards instead. + +To reach them, open Grafana through the monitoring hub, browse to the `hubble` folder in the dashboard browser, and pick a dashboard. + +## Metrics + +Hubble exposes the following, all queryable directly in Grafana: + +- `hubble_flows_processed_total` — total number of flows processed +- `hubble_dns_queries_total` — DNS queries by type +- `hubble_dns_responses_total` — DNS responses by status +- `hubble_drop_total` — dropped packets by reason +- `hubble_tcp_flags_total` — TCP connections by flag +- `hubble_http_requests_total` — HTTP requests by method and status + +## Troubleshooting + +Check that Relay and the UI are running: + +```bash +kubectl get pods -n cozy-cilium -l k8s-app=hubble-relay +kubectl get pods -n cozy-cilium -l k8s-app=hubble-ui +``` + +Verify the metrics endpoint is serving: + +```bash +kubectl port-forward -n cozy-cilium svc/hubble-metrics 9965:9965 +curl http://localhost:9965/metrics +``` + +Confirm the scrape target exists — if the dashboards are empty but the endpoint above returns data, this is usually the missing link: + +```bash +kubectl get servicemonitor -n cozy-cilium +``` diff --git a/content/en/docs/next/networking/vm-external-vlan.md b/content/en/docs/next/networking/vm-external-vlan.md new file mode 100644 index 00000000..f4cf907b --- /dev/null +++ b/content/en/docs/next/networking/vm-external-vlan.md @@ -0,0 +1,130 @@ +--- +title: "Attaching a Virtual Machine to an External VLAN" +linkTitle: "VM External VLAN" +description: "Bridge a virtual machine onto a physically-routed VLAN so it shares a broadcast domain with external hardware, and why macvlan cannot work for this." +weight: 35 +--- + +This page describes how to attach a Cozystack virtual machine (the [`vm-instance`](/docs/next/virtualization/vm-instance/) application) directly to an external, physically-routed VLAN — the layer-2 segment a VM needs when it must appear on the same broadcast domain as external hardware (a licensing appliance, a storage box, a gateway managed outside the cluster), with an address from that VLAN's subnet rather than from the cluster overlay. + +The default Cozystack VM networking is overlay-only (the pod network, plus optional KubeOVN [VPC subnets](/docs/next/networking/vpc/)). Bridging a VM onto a real VLAN is a different pattern and has one non-obvious constraint: **it works with a Linux bridge and the `bridge` CNI plugin, and it does not work with `macvlan`.** The rest of this guide explains why and gives a working recipe. + +## Why `bridge` and not `macvlan` + +KubeVirt attaches a VM interface to a secondary network using **bridge binding** by default (this is what the `vm-instance` chart emits for every network in `.spec.networks`). With bridge binding the guest's own MAC address is placed on the wire — the launcher pod does not masquerade or translate it. + +A `macvlan` attachment is incompatible with that model. `macvlan` demultiplexes inbound frames strictly by the MAC address of the macvlan child interface. Because KubeVirt puts the *guest's* MAC on the wire — not the macvlan child's — replies from the gateway or other hosts arrive at the parent interface addressed to the guest MAC, do not match any macvlan child, and are silently dropped before they ever reach the VM. The symptom is a guest that can transmit (ARP requests and pings leave, visible in `tcpdump` on the parent interface) but never receives a reply (its neighbor entry for the gateway stays `FAILED`). As a secondary consequence, the host cannot reach macvlan children through the parent interface either, so a host-side service on the parent IP is unreachable from the VMs. + +A **Linux bridge** does not have this limitation: it forwards by learned MAC on all bridged ports, so the guest MAC is reachable, and the host can carry an address on the bridge itself to talk to the VMs. Attach the VLAN sub-interface to a bridge and point a `bridge`-type NetworkAttachmentDefinition at it. + +## Overview + +Three pieces cooperate: + +1. A **Linux bridge on each node** that enslaves the tagged VLAN sub-interface. This is node-level networking — it is configured by your node provisioning (netplan / Talos machine config / systemd-networkd), not by a Cozystack chart. +2. A **`NetworkAttachmentDefinition`** of type `bridge` referencing that bridge, created in the VM's tenant namespace. +3. The **`vm-instance`** application referencing the NetworkAttachmentDefinition by name in `.networks`, with the guest's static address supplied through cloud-init. + +## Prerequisites + +- The `multus` package is enabled (it provides the `NetworkAttachmentDefinition` CRD and the secondary-network plumbing). +- The `bridge` CNI plugin is present in `/opt/cni/bin` on every node. The `multus` package puts it there itself on every platform; see [the multus package README](https://github.com/cozystack/cozystack/blob/main/packages/system/multus/README.md) for what it stages and the opt-out, and read it before upgrading a cluster whose `/opt/cni/bin` you provision yourself. Verify with `ls /opt/cni/bin/bridge`; a missing binary makes the NetworkAttachmentDefinition fail with `failed to find plugin "bridge" in path [/opt/cni/bin]`. +- There is no IPAM plugin in this path — addresses are assigned inside the guest, not by the CNI. Plan static addresses per VM. + +## 1. Linux bridge on the node + +Create a bridge that enslaves the tagged VLAN sub-interface. The VLAN sub-interface itself carries no address; the bridge carries the host's presence on that VLAN (optional, but useful for a gateway-reachability sanity path and for any host-side service the VMs must reach). + +This example uses netplan on an Ubuntu/Debian node; the VLAN id and subnet are illustrative (`203.0.113.0/24`, VLAN 100, gateway `203.0.113.1`). Adapt to your uplink naming and to Talos or `systemd-networkd` if that is your provisioning: + +```yaml +network: + version: 2 + vlans: + # Tagged VLAN sub-interface, no address of its own — enslaved to the bridge. + uplink.100: + id: 100 + link: uplink + bridges: + br100: + interfaces: + - uplink.100 + # Optional host presence on the VLAN. Keep the node's default route on + # its management interface — do not add a default route here. + addresses: + - 203.0.113.2/24 +``` + +Notes: + +- The node's **default route must stay on the management interface.** The bridge address (if any) is only for on-VLAN reachability, not a second default gateway. +- `netplan apply` cannot move an interface into a bridge while a consumer still holds it (for example a `virt-launcher` pod using a previous `macvlan` attachment). Remove the consumer first (delete the VMI so the launcher releases the interface), then reconfigure. +- After a reboot, `systemd-networkd` may briefly report the bridge "routable" while the link is not yet actually up. If your VMs need the VLAN immediately at boot, gate their start on a reachability check, or re-run `netplan apply` until the gateway answers. + +## 2. NetworkAttachmentDefinition + +Create a `bridge`-type NetworkAttachmentDefinition in the tenant namespace that will host the VM. The `vm-instance` chart resolves a network by name **in the VM's own namespace**, so one copy must exist in every tenant namespace that runs VMs on this VLAN. + +```yaml +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: vlan100 + namespace: tenant-example +spec: + config: | + { + "cniVersion": "0.3.1", + "type": "bridge", + "bridge": "br100", + "ipam": {} + } +``` + +- `bridge` must match the bridge name from step 1 (`br100` here). +- `ipam: {}` — no cluster-side address assignment; the guest configures its address itself (step 3). + +## 3. Attach the VM and assign a static address + +Reference the NetworkAttachmentDefinition by name in the `vm-instance` values. Because the chart does not support `networkData`, the static address goes into cloud-init `userData` (`cloudInit`), written by the guest at first boot: + +```yaml +# vm-instance values +instanceType: u1.medium +instanceProfile: ubuntu +disks: + - name: example-system +networks: + - name: vlan100 +cloudInit: | + #cloud-config + write_files: + - path: /etc/netplan/60-vlan100.yaml + permissions: "0600" + content: | + network: + version: 2 + ethernets: + # Match the second NIC (the pod-network NIC is the first). Use the + # interface that comes up without a DHCP lease. + enp2s0: + addresses: + - 203.0.113.10/24 + runcmd: + - netplan apply +``` + +The VM ends up with two interfaces: the always-present **pod-network** NIC (`default`, used for cluster-internal traffic and for the `vm-instance` external-access features) and the **VLAN** NIC. The `/24` address above brings up only the connected route for the VLAN subnet — it adds no default route, so the guest's egress stays wherever you want it (typically the pod NIC). If the VLAN is meant to be the guest's default gateway instead, add a default route under the VLAN NIC and remove it from the pod NIC. + +## Gotchas + +- **VMs are dual-homed.** The `vm-instance` chart always adds the pod-network NIC in addition to any `networks` you declare; there is no single-homed (VLAN-only) option today. Address the VLAN NIC inside the guest and leave the pod NIC to the cluster. +- **No `networkData`.** The chart wires cloud-init through `userData` only, so in-guest static configuration (netplan `write_files` plus `netplan apply`, as above) is the way to assign the VLAN address. +- **MAC changes on VM re-creation.** KubeVirt generates a fresh guest MAC each time the VM object is re-created, and `vm-instance` exposes no way to pin it, so re-creating a VM changes its MAC. The upstream gateway then holds a stale ARP entry for the old MAC for a few minutes, so "gateway unreachable" immediately after re-creating a VM is expected — wait for the ARP entry to age out (roughly five minutes) rather than treating it as a fault. +- **Host-to-VM traffic.** If the host must talk to the VMs (a proxy, a health check), give the bridge a host address on the VLAN (step 1) — traffic through a bare VLAN sub-interface to bridge-attached guests will not work the way `macvlan` users expect. + +## See also + +- [Attaching GPUs to virtual machines](/docs/next/virtualization/gpu/) — passing NVIDIA GPUs and vGPU profiles into the same VMs. +- [Networking architecture](/docs/next/networking/architecture/) — how the default overlay data plane is put together. +- KubeVirt user guide, [Interfaces and Networks](https://kubevirt.io/user-guide/network/interfaces_and_networks/) — bridge binding versus other binding methods. diff --git a/content/en/docs/next/operations/troubleshooting/gpu-operator-host-driver.md b/content/en/docs/next/operations/troubleshooting/gpu-operator-host-driver.md new file mode 100644 index 00000000..ba035318 --- /dev/null +++ b/content/en/docs/next/operations/troubleshooting/gpu-operator-host-driver.md @@ -0,0 +1,147 @@ +--- +title: "GPU Passthrough Fails on a Host with a Pre-installed NVIDIA Driver" +linkTitle: "GPU Operator: host driver" +description: "Why vfio-manager refuses to bind vfio-pci when the node already has an apt-installed NVIDIA driver, and how to recover the node." +weight: 30 +--- + +The `default` (passthrough) variant of the `cozystack.gpu-operator` package assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**. On a node that already has the NVIDIA driver installed through the distro package manager, the operator detects it, declines to touch it, and the passthrough setup never completes. + +Verified 2026-05-28 against the `gpu-operator` chart v26.3.1 with `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`, on an Ubuntu 24.04 host carrying `nvidia-driver-580-open` 580.82.07. + +{{< note >}} + +If you want GPUs in **containers** rather than in VMs, you do not need this recovery at all — use the [`container` variant](/docs/next/operations/gpu-container-workloads/), which is designed for exactly this host shape and keeps the host driver in place. + +{{< /note >}} + +## Symptom + +`kubectl get pods -n cozy-gpu-operator` shows `nvidia-vfio-manager-*` stuck in `Init:Error` or `Init:CrashLoopBackOff` — the init container exits non-zero after detecting the host driver, so kubelet keeps restarting it. + +Its log shows it skipped the bind step: + +```text +Host driver detected: 580.82.07 +NVIDIA GPU driver is already pre-installed on the node, + disabling the containerized driver +Labeling node with nvidia.com/gpu.deploy.driver=pre-installed +``` + +`nvidia-sandbox-validator` then crashloops: + +```text +Error: error validating vfio-pci driver installation: + device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia' +``` + +`lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and this reports `{}` — no GPU resource was registered: + +```bash +kubectl get node -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))' +``` + +## Why it happens + +The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand. That path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present". File existence is not pre-tested — if `nvidia-smi` is missing the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed. + +On a positive detection the binary logs `Host driver detected: `, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, and exits. + +**`FORCE_REINSTALL` does not bypass this.** `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env and flag pair, but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top. Operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself. + +## Recovery: clean the host + +Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. + +Two pitfalls to avoid. `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, so a blanket autoremove can take RDMA out on a converged GPU plus RDMA host. And a hardcoded `apt purge 'nvidia-*' 'cuda-*'` pattern list is fragile — `apt` treats `*` as a cache-wide regex and **aborts the entire transaction, purging nothing**, if any pattern matches nothing in the cache (for example `cuda-*` on a host without NVIDIA's CUDA repo). Build the list from what is actually installed instead: + +```bash +# dpkg-query patterns are true globs over INSTALLED packages: no +# zero-match abort (unlike apt's cache-wide regex) and no accidental +# substring over-match. List first, then review before purging. +dpkg-query -W -f '${Package}\n' 'nvidia-*' 'libnvidia-*' 'cuda-*' 2>/dev/null +``` + +Review the list before purging: + +- `nvidia-dkms-*` and `nvidia-kernel-*` are the load-bearing kernel pieces — without removing them DKMS rebuilds `nvidia.ko` on the next reboot and the blacklist below is bypassed by any explicit `modprobe`. +- On a **converged GPU plus RDMA host**, drop any `libnvidia-*` that belong to NVIDIA DOCA / Mellanox OFED — purging those breaks RDMA. +- If the list is **empty**, the driver was installed with NVIDIA's `.run` installer rather than apt — run `sudo nvidia-uninstall` instead of the purge below. + +Then purge the reviewed list, blacklist the modules, and rebuild the initramfs: + +```bash +# Replace with the packages you kept from the list above. +sudo apt purge nvidia-driver-580-open nvidia-dkms-580-open <...> + +sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF' +blacklist nouveau +blacklist nvidia +blacklist nvidia_drm +blacklist nvidia_modeset +blacklist nvidia_uvm +blacklist nvidia_peermem +EOF + +# -k all rebuilds EVERY installed kernel's initramfs (plain -u touches +# only the running kernel), so a just-upgraded kernel also boots with +# the blacklist and cannot re-claim the GPU. +sudo update-initramfs -u -k all +sudo reboot +``` + +## Confirm the host is clean + +The init container exits non-zero after detecting a host driver, so both DaemonSet pods sit in `Init:CrashLoopBackOff` and **will retry on their own** — deleting them just skips the up-to-five-minute backoff window. + +```bash +# Should print nothing. +lsmod | grep -E '^(nvidia|nouveau)' + +# command -v matches what isHostDriver does (PATH lookup inside the +# chroot), so it also catches /usr/local/bin/nvidia-smi left by a .run +# or CUDA-toolkit install. Prints "ok: gone" on success. +command -v nvidia-smi >/dev/null && echo "STILL PRESENT — purge incomplete" || echo "ok: gone" + +# A leftover DKMS module rebuilds nvidia.ko on the next kernel update +# and an explicit modprobe bypasses the blacklist — should print nothing. +dkms status | grep -i nvidia + +# Skip the CrashLoopBackOff backoff window by deleting the stuck pods. +# The DaemonSet labels are operator-managed and not stable across +# gpu-operator versions, so delete by name pattern — anchored so the +# match is the two DaemonSets and nothing else. +kubectl -n cozy-gpu-operator get pods -o name \ + | grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \ + | xargs -r kubectl -n cozy-gpu-operator delete +``` + +Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource: + +```bash +lspci -nnk -d 10de: | grep 'Kernel driver in use' +# Kernel driver in use: vfio-pci + +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com +``` + +## Clear the leftover label + +One label is **not** cleaned up automatically. The init container set `nvidia.com/gpu.deploy.driver=pre-installed`, and the operator's success path restores only the operand labels — `rescheduleGPUOperatorComponents` in `k8s-driver-manager` v0.10.0 touches the validator, toolkit and device-plugin labels, not `deploy.driver`. So the node keeps `pre-installed` indefinitely: it is the same label the Symptom section uses as evidence, and it would disable the containerized-driver DaemonSet if the node later switches to a container workload. + +```bash +# Nothing on the success path resets this label; clear it so it does not +# mislead future debugging or block a later container-workload switch. +kubectl label node nvidia.com/gpu.deploy.driver- +``` + +## Known limitation + +The skip-on-pre-installed behaviour lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. + +Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads therefore cannot share the same GPU with the passthrough variant. Two paths out: + +- **Use the [`container` variant](/docs/next/operations/gpu-container-workloads/)** if the workloads can be containers rather than VMs. It targets the apt-installed-driver host shape and exposes GPUs to pods without unbinding the host driver, so no purge is needed. +- **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191), still open. + +Talos is unaffected: the Talos image ships only the `vfio-pci` extension and no host NVIDIA stack, so the clean-host check passes trivially. This page applies to distributions where you installed the host driver yourself — typically Ubuntu, Debian or RHEL with `apt install nvidia-driver-*` or the equivalent. diff --git a/content/en/docs/next/virtualization/gpu.md b/content/en/docs/next/virtualization/gpu.md index dbebacfa..8c6f7c04 100644 --- a/content/en/docs/next/virtualization/gpu.md +++ b/content/en/docs/next/virtualization/gpu.md @@ -270,7 +270,7 @@ GPU passthrough assigns an entire physical GPU to a single VM. To share one GPU ### vGPU (Virtual GPU) -NVIDIA vGPU uses mediated devices (mdev) to create virtual GPUs assignable to VMs. This is the only production-ready solution for GPU sharing between VMs. +NVIDIA vGPU is the only production-ready solution for GPU sharing between VMs. Two host-side models exist depending on GPU generation: mediated devices (mdev) on Pascal through Ampere, and SR-IOV virtual functions on Ada Lovelace and newer. See [NVIDIA vGPU for virtual machines](/docs/next/virtualization/vgpu/) for the full setup, including profile assignment and DLS licensing. **Requirements:** - NVIDIA vGPU license (commercial, purchased from NVIDIA) diff --git a/content/en/docs/next/virtualization/vgpu.md b/content/en/docs/next/virtualization/vgpu.md new file mode 100644 index 00000000..3002bd83 --- /dev/null +++ b/content/en/docs/next/virtualization/vgpu.md @@ -0,0 +1,292 @@ +--- +title: "NVIDIA vGPU for Virtual Machines" +linkTitle: "vGPU" +description: "Slice one physical NVIDIA GPU across several virtual machines with the vgpu variant of the GPU Operator package, including SR-IOV profile assignment and DLS licensing." +weight: 45 +--- + +This page describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines. For handing a whole GPU to one VM, see [GPU passthrough](/docs/next/virtualization/gpu/); for GPUs in containers rather than VMs, see [containerized GPU workloads](/docs/next/operations/gpu-container-workloads/). + +Verified 2026-04-29 against KubeVirt `main` (`virt-handler` nightly `20260429_74d7c52588`), the `vgpu` variant of `cozystack.gpu-operator`, the NVIDIA vGPU 20.0 host driver `595.58.02` and GRID guest driver `595.58.03`. + +## Two driver models + +NVIDIA's vGPU driver uses two different host-side models depending on GPU generation: + +- **Mediated devices (mdev)** — Pascal / Volta / Turing / Ampere up to A100 and A30. The driver creates `mdev` parent devices under `/sys/class/mdev_bus/`; KubeVirt advertises them via `permittedHostDevices.mediatedDevices`. +- **SR-IOV with per-VF sysfs** — Ada Lovelace (L4, L40, L40S, …) and Blackwell (B100, …) on the vGPU 17/20 driver branch. The driver creates SR-IOV virtual functions; profile selection happens via `/sys/bus/pci/devices//nvidia/current_vgpu_type`. KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` after [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890). + +This guide focuses on the **SR-IOV path**, which is the only model NVIDIA supports for current data-centre GPUs. Mdev is mentioned for completeness; for Pascal to Ampere refer to the upstream NVIDIA GPU Operator documentation. + +## Prerequisites + +- An Ada Lovelace or newer NVIDIA GPU that supports SR-IOV vGPU (L4, L40, L40S, and similar). +- Ubuntu 24.04 host OS. Older Ubuntu releases also work if the upstream `gpu-driver-container` repository has a matching `vgpu-manager/` Dockerfile. **Talos Linux is not recommended** for vGPU: NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero [closed siderolabs/extensions#461](https://github.com/siderolabs/extensions/issues/461) noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that includes the driver in-tree is therefore not feasible without a private fork that violates the EULA. +- KubeVirt with [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890) ("vGPU: SRIOV support", merged to `main` 2026-04-10). Targeted at the next minor release (v1.9.0); track the pull request for the actual release tag. Released tags up to and including v1.8.x do not include the patch and backports are not planned. If you need vGPU before v1.9.0 lands you have to run a `main`-based nightly build of `virt-handler`; the rest of the operator can stay on the latest released tag. +- An NVIDIA vGPU Software or NVIDIA AI Enterprise subscription (the `.run` is not redistributable). +- A reachable NVIDIA Delegated License Service (DLS) instance and a matching `client_configuration_token.tok` file. + +## Variants + +The `gpu-operator` package exposes three variants. This page is vGPU-focused; the variant inventory is shared. + +- **`default`** — passthrough mode (`vfio-pci`). The whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, so no proprietary driver is needed on the host. On a host that already carries an apt-installed NVIDIA driver this variant will not complete — see [GPU passthrough fails on a host with a pre-installed NVIDIA driver](/docs/next/operations/troubleshooting/gpu-operator-host-driver/). +- **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU. +- **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin, on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit`. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. See [containerized GPU workloads](/docs/next/operations/gpu-container-workloads/). + +## Building the vGPU Manager image + +The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager//` and is the source of truth for build arguments, base images and supported OS releases. Follow the README in that repository. + +The proprietary `.run` is the **Linux KVM** variant, not the Ubuntu KVM `.deb` (which ships pre-built modules for stock kernels only). It comes from the [NVIDIA Licensing Portal](https://ui.licensing.nvidia.com) under an NVIDIA AI Enterprise or vGPU subscription. + +{{< warning >}} + +**EULA:** never push the resulting image to a publicly readable registry. Use a private registry — an in-cluster Harbor works well as a non-proxy project. + +{{< /warning >}} + +## Deploying with the vgpu variant + +The platform's `iaas` bundle deploys the gpu-operator Package CR when `cozystack.gpu-operator` is in `bundles.enabledPackages` and `bundles.iaas.gpuOperatorVariant: vgpu` is set. The vGPU Manager image is proprietary and not redistributable, so the bundle does not ship a default tag — build the container per the upstream `gpu-driver-container` recipe and supply the private-registry coordinates through platform values: + +```yaml +bundles: + iaas: + enabled: true + gpuOperatorVariant: vgpu + enabledPackages: + - cozystack.gpu-operator + +gpu: + vgpuManager: + repository: registry.example.com/nvidia + image: vgpu-manager + version: "595.58.02-ubuntu24.04" + # imagePullSecrets lives per-component (vgpuManager, driver, + # validator, dcgmExporter, …). The value is a list of strings, + # not [{name: ...}]. + imagePullSecrets: + - nvidia-registry-secret +``` + +The platform forwards `gpu.vgpuManager` into the emitted gpu-operator Package CR's `components.gpu-operator.values.gpu-operator.vgpuManager`, so the bundle handles the variant and image coordinates in one place. If you need to override anything else on the gpu-operator chart (driver, validator, dcgmExporter, custom node selectors), hand-craft a `Package` CR named `cozystack.gpu-operator` with the full `components.gpu-operator.values` block — that takes precedence over the bundle render. + +The `nvidia-registry-secret` should be a docker-registry Secret created beforehand in `cozy-gpu-operator`. + +Verify the DaemonSet is running and `nvidia.ko` loads on every GPU node: + +```bash +kubectl -n cozy-gpu-operator get pods -l app=nvidia-vgpu-manager-daemonset +kubectl -n cozy-gpu-operator exec -it -- nvidia-smi +``` + +`nvidia-smi` should enumerate the physical GPUs and report `Host VGPU Mode : SR-IOV`. + +## Profile assignment (SR-IOV path) + +{{< caution >}} + +**The `vgpu` variant is experimental on Ada and newer, and ships without a profile-assignment loop.** NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada and newer — the DaemonSet errors with "no parent devices found for GPU at index '0'" and is therefore disabled by default in `values-vgpu.yaml`. Until an SR-IOV-aware controller ships, profile assignment is an out-of-band step that must be re-applied after every node reboot (`current_vgpu_type` resets to 0 on PCIe re-enumeration). Without this step `permittedHostDevices.pciHostDevices` reports zero allocatable resources and no VM can request the vGPU. **Do not deploy the `vgpu` variant in production until you have an automated profile-assignment mechanism in place** — typically a small DaemonSet that reads a ConfigMap (` = `) and writes the corresponding `current_vgpu_type` files at boot. + +{{< /caution >}} + +Once `nvidia.ko` is loaded the driver enables SR-IOV (16 VFs per L40S by default). Each VF needs a vGPU profile written to its sysfs: + +```bash +# from inside the nvidia-vgpu-manager-daemonset pod (privileged, hostPID) +echo 1155 > /sys/bus/pci/devices/0000:02:00.5/nvidia/current_vgpu_type +``` + +The numeric profile ID can be discovered per-VF: + +```bash +cat /sys/bus/pci/devices/0000:02:00.5/nvidia/creatable_vgpu_types +``` + +For Pascal to Ampere GPUs (V100, T4, A100, A30) the mdev model still applies. Flip `vgpuDeviceManager.enabled: true` in your Package CR overrides — NVIDIA's device manager works correctly there. + +## KubeVirt configuration + +When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step. + +If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly — typically to apply overrides the bundle does not expose — the platform does **not** auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow you also hand-craft a `cozystack.kubevirt` Package CR with `components.kubevirt.values.extraFeatureGates: [HostDevices]` and the appropriate `permittedHostDevices` block. The escape-hatch values shape under `.gpu` below is documented for the bundle-managed flow only; the manual Package-CR override path takes precedence over the bundle render whenever both exist. + +- `developerConfiguration.featureGates` gets `HostDevices` appended (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `spec.template.spec.domain.devices.hostDevices` without it). +- `permittedHostDevices.pciHostDevices` is filled from `packages/core/platform/files/gpu-passthrough-defaults.yaml` in the [cozystack repository](https://github.com/cozystack/cozystack) when `bundles.iaas.gpuOperatorVariant: default` (the package default). The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and Volta (V100/V100S). All entries carry `externalResourceProvider: true` because the resource names come from `nvidia-sandbox-device-plugin`, not from KubeVirt's in-tree device plugin. +- `permittedHostDevices.mediatedDevices` is filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu`. This list only *exposes*, by profile name (`mdevNameSelector`), mdevs that the GPU Operator's vGPU Device Manager *creates* on the node; the platform does not ship a numeric `mediatedDevicesConfiguration` default (those `nvidia-NNN` type ids are per-SKU and per-driver sysfs indices with no portable value — set `.gpu.mediatedDevicesConfiguration` yourself, with host-verified ids, only if you want KubeVirt rather than the Device Manager to create mdevs). The starter set covers Pascal to Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and Blackwell SR-IOV vGPU are out of scope for the chart's default list; advertise those VFs via the user-override hook below. + +### Extending or replacing the default table + +The platform exposes three knobs under `.gpu`: + +```yaml +gpu: + # Extend the platform defaults with cluster-specific entries. Both list + # keys are read in both variants: pciHostDevices feeds the passthrough + # (vfio-pci) path AND the post-kubevirt#16890 SR-IOV vGPU VF path on + # Ada Lovelace / Blackwell; mediatedDevices feeds the pre-#16890 mdev + # path on Pascal–Ampere. Both render into the same KubeVirt CR. + permittedHostDevices: + pciHostDevices: + - pciVendorSelector: "10DE:26B9" # L40S, advertised as a VF for SR-IOV vGPU + resourceName: nvidia.com/L40S-24Q + # externalResourceProvider is intentionally omitted here: after + # kubevirt/kubevirt#16890, virt-handler's in-tree device plugin + # advertises the resource directly, no sandbox plugin in the loop. + mediatedDevices: [] + # mediatedDevicesConfiguration makes KubeVirt itself create mdevs (vgpu + # mode). No platform default: mdev creation is normally delegated to the + # vGPU Device Manager (name-based), and these mediatedDeviceTypes are + # host/driver-specific nvidia-NNN sysfs indices (look yours up via + # /sys/bus/pci/devices//mdev_supported_types/*/name). Set this only + # to opt into KubeVirt-driven creation; mergeOverwrite REPLACES a + # supplied top-level key wholesale. + mediatedDevicesConfiguration: {} + # Wipe the platform defaults entirely and ship only the cluster's + # curated lists. Useful for non-NVIDIA-only clusters and strict + # allowlist requirements. + replaceDefaults: false +``` + +`replaceDefaults: false` (the default) appends user entries to the NVIDIA defaults. `replaceDefaults: true` drops the NVIDIA table entirely — if you do not then supply your own `pciHostDevices` or `mediatedDevices` list, the rendered KubeVirt CR has no `permittedHostDevices` block and the admission webhook rejects every GPU VM. + +### Resource names from `nvidia-sandbox-device-plugin` + +The `resourceName` strings in `gpu-passthrough-defaults.yaml` are what `nvidia-sandbox-device-plugin` (`nvcr.io/nvidia/kubevirt-gpu-device-plugin`) advertises: it derives each slug mechanically from the device's PCI-IDs database name by uppercasing it, turning `/`, `.` and whitespace into `_`, and stripping the remaining non-alphanumerics (the `[` and `]`). So `TU104GL [Tesla T4]` becomes `nvidia.com/TU104GL_TESLA_T4` and `GA100GL [A30 PCIe]` becomes `nvidia.com/GA100GL_A30_PCIE` — the slug carries every token the PCI-IDs string holds (the `GL` die suffix, the `Tesla` brand on Turing and Volta, form factor, memory), not a tidy `_`. The names track the pci.ids snapshot bundled in the plugin image, so a different plugin build can publish a different string — check with `kubectl describe node | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor and device IDs themselves are stable across driver versions. + +### SR-IOV PF versus VF on Ada Lovelace and newer + +On L40S and other Ada Lovelace cards the SR-IOV VFs report the same PCI device ID as the PF — `lspci -nn -d 10de:` on the host shows both as `[10de:26b9]`. `virt-handler` distinguishes them by "is a VF and has a vGPU profile", so a single `pciVendorSelector` matches the right set. Verify on your specific GPU before assuming this — some other generations split PF and VF IDs. + +`externalResourceProvider: true` is **not** required when the resource is advertised by `virt-handler`'s in-tree device plugin (the SR-IOV path after kubevirt#16890). The platform passthrough defaults include the flag because that path is driven by the external sandbox plugin. + +### Verifying allocatable capacity + +```bash +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable.nvidia\.com/L40S-24Q}{"\n"}{end}' +``` + +## Licensing (DLS) + +vGPU 17/20 uses the NVIDIA Delegated License Service. The legacy `ServerAddress=` and `ServerPort=7070` lines in `gridd.conf` are no longer authoritative — `nvidia-gridd`, running **inside the guest**, reads the DLS endpoint from the ClientConfigToken file directly. + +The host vGPU Manager DaemonSet does not request a license — it only enables SR-IOV and loads `nvidia.ko`. Licensing is consumed entirely by the guest. The gpu-operator chart's `driver.licensingConfig.secretName` would mount the Secret into the **driver pod on the host**, where it has no effect for SR-IOV vGPU; do not wire the licensing Secret through it. + +Instead, deliver the token and `gridd.conf` to the guest via cloud-init or a containerDisk overlay: + +```yaml +# inside the VirtualMachine cloudInitNoCloud userData +write_files: +- path: /etc/nvidia/ClientConfigToken/client_configuration_token.tok + # 0744 follows NVIDIA's recommendation in the Virtual GPU Software + # Licensing User Guide ("Configuring a Licensed Client on Linux"): + # nvidia-gridd does not necessarily run as the file owner. + # https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/ + permissions: '0744' + encoding: b64 + content: +- path: /etc/nvidia/gridd.conf + permissions: '0644' + content: | + # FeatureType selects which vGPU Software license the guest requests. + # 0 — unlicensed state (no license requested; Q profiles run in + # reduced mode after the grace period). + # 1 — NVIDIA vGPU. The driver auto-selects the correct license + # type from the configured vGPU profile (Q → vWS, B → vPC, + # A → vCS / Compute). Use this for SR-IOV vGPU profiles. + # 2 — explicitly NVIDIA RTX Virtual Workstation. + # 4 — explicitly NVIDIA Virtual Compute Server. + FeatureType=1 +``` + +Verify activation inside the guest: + +```bash +nvidia-smi -q | grep 'License Status' +# License Status : Licensed +``` + +If the guest reports `Unlicensed (Unrestricted)` for more than a couple of minutes, check `journalctl _COMM=nvidia-gridd` for handshake errors against the DLS endpoint baked into the token. + +### Migrating from chart v25.x + +Upstream deprecated `driver.licensingConfig.configMapName` in favour of `driver.licensingConfig.secretName`. The old key still works but emits a deprecation warning at render time. If your existing `Package` CR set the licensing reference via `configMapName`, switch it to `secretName` on this upgrade — the Secret content (`gridd.conf` and the ClientConfigToken) does not need to change. This applies to passthrough deployments that drove host-side licensing through the gpu-operator chart; SR-IOV vGPU does not consume the host-side licensing knob at all, as above. + +## Sample VirtualMachine + +Either `hostDevices` or `gpus` accepts the resource (the upstream KubeVirt API resolves both PCI and mediated-device pools), but the convention is to use `hostDevices` for VF-style PCI passthrough: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: vgpu-smoke + namespace: tenant-example +spec: + runStrategy: Always + template: + spec: + domain: + cpu: + cores: 4 + memory: + guest: 8Gi + devices: + disks: + - name: rootdisk + disk: + bus: virtio + interfaces: + - name: default + masquerade: {} + hostDevices: + - name: gpu0 + deviceName: nvidia.com/L40S-24Q + networks: + - name: default + pod: {} + volumes: + - name: rootdisk + # A 2.4 GiB containerDisk overlay is too small to install + # the GRID guest driver in-place. Use a CDI DataVolume of + # 20 GiB+ in production. + containerDisk: + image: quay.io/containerdisks/ubuntu:24.04 +``` + +Inside the guest, install the GRID driver from the `.run` — the GUEST `.run`, distinct from the host `vgpu-kvm` package — after which `nvidia-smi` should report the configured profile: + +```text +| 0 NVIDIA L40S-24Q Off | 00000000:0E:00.0 Off | 0 | +| 17 MiB / 24576 MiB P0 Default | +``` + +## Profile reference (L40S) + +L40S supports the full Q (RTX vWS), B (vPC) and A (vCS / Compute) profile families. The numeric IDs come from the driver and are visible in `creatable_vgpu_types`: + +| Profile | Frame Buffer | Max instances per L40S | Use case | +| --- | --- | --- | --- | +| L40S-1Q | 1 GB | 48 | Light 3D / VDI | +| L40S-2Q | 2 GB | 24 | Medium 3D / VDI | +| L40S-4Q | 4 GB | 12 | Heavy 3D / VDI | +| L40S-6Q | 6 GB | 8 | Professional 3D | +| L40S-8Q | 8 GB | 6 | AI / ML inference | +| L40S-12Q | 12 GB | 4 | AI / ML training | +| L40S-24Q | 24 GB | 2 | Large AI workloads | +| L40S-48Q | 48 GB | 1 | Full GPU equivalent | + +Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Documentation](https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/). + +## OS support summary + +The `container` column assumes the host already ships the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager, with the `nvidia` runtime registered in containerd. With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override. Talos installs the driver under a non-standard prefix, so the operator does not find it at the default location — see `packages/system/gpu-operator/examples/` in the [cozystack repository](https://github.com/cozystack/cozystack) for the Talos-specific path with a compat DaemonSet and an explicit `hostPaths.driverInstallDir` override. + +| Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) | +| --- | --- | --- | --- | +| Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see [host-driver recovery](/docs/next/operations/troubleshooting/gpu-operator-host-driver/) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver plus nvidia-container-toolkit | +| Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | +| Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | +| Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus an `nvidia-driver` patch for usr-merge (details pending) | ⚠️ same patch plus own Dockerfile fork | ✅ | +| Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run` | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` as a starting point | diff --git a/content/en/docs/v1.6/networking/architecture.md b/content/en/docs/v1.6/networking/architecture.md index f61ca299..293da6c8 100644 --- a/content/en/docs/v1.6/networking/architecture.md +++ b/content/en/docs/v1.6/networking/architecture.md @@ -433,7 +433,7 @@ cilium: enabled: true ``` -See [Enabling Hubble](https://docs.cilium.io/en/stable/observability/hubble/) for full configuration details. +See [Enabling Hubble for network observability](/docs/v1.6/networking/hubble/) for the metrics list this needs, the Grafana dashboards Cozystack ships for it, and troubleshooting. The upstream [Cilium Hubble documentation](https://docs.cilium.io/en/stable/observability/hubble/) covers the remaining knobs. ## Traffic Flow Summary diff --git a/content/en/docs/v1.6/networking/hubble.md b/content/en/docs/v1.6/networking/hubble.md new file mode 100644 index 00000000..6a56c37f --- /dev/null +++ b/content/en/docs/v1.6/networking/hubble.md @@ -0,0 +1,96 @@ +--- +title: "Enabling Hubble for Network Observability" +linkTitle: "Hubble" +description: "Turn on Cilium's Hubble observability stack, and read the flow, DNS and L7 metrics through the platform Grafana dashboards Cozystack ships for it." +weight: 50 +--- + +Hubble is the network and security observability layer built on top of Cilium. It gives visibility into the communication and behaviour of services in the cluster — flow logs, DNS queries, and L7 request metrics. + +Hubble is **disabled by default** in Cozystack to keep resource usage down. This page covers turning it on and reading the results. For where Hubble sits in the data plane, see [Networking architecture](/docs/v1.6/networking/architecture/#observability-with-hubble). + +## Prerequisites + +- A Cozystack cluster running Cilium as the CNI (the default). +- The [Monitoring](/docs/v1.6/operations/services/monitoring/) hub deployed, for Grafana access and metric storage. + +## Enable Hubble + +Enable Hubble, Relay and the UI in the Cilium configuration, and turn on the metrics you want exported: + +```yaml +cilium: + hubble: + enabled: true + relay: + enabled: true + ui: + enabled: true + metrics: + enabled: + - dns + - drop + - tcp + - flow + - port-distribution + - icmp + - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction +``` + +The `metrics.enabled` list is what makes the dashboards below work — without it Hubble runs but exports nothing for Grafana to draw. The `httpV2` entry in particular must keep its `labelsContext`, because the L7 HTTP dashboard groups by source and destination workload and cannot do so if those labels are absent. + +### Components + +Enabling Hubble brings up: + +- **Hubble Relay** — aggregates flow data from all Cilium agents. +- **Hubble UI** — web interface for exploring network flows. +- **Hubble Metrics** — Prometheus metrics for network observability. + +## Grafana dashboards + +Cozystack ships four Hubble dashboards, delivered in the `hubble` folder of the platform Grafana: + +| Dashboard | Description | +|-----------|-------------| +| **Overview** | General Hubble metrics including processing statistics | +| **DNS Namespace** | DNS query and response metrics by namespace | +| **L7 HTTP Metrics** | HTTP layer 7 metrics by workload | +| **Network Overview** | Network flow overview by namespace | + +These are infrastructure dashboards, so they are provisioned only for the platform-level Monitoring release — the one in `tenant-root` or `cozy-monitoring`. A tenant's own Grafana does not receive them; tenants see their own application dashboards instead. + +To reach them, open Grafana through the monitoring hub, browse to the `hubble` folder in the dashboard browser, and pick a dashboard. + +## Metrics + +Hubble exposes the following, all queryable directly in Grafana: + +- `hubble_flows_processed_total` — total number of flows processed +- `hubble_dns_queries_total` — DNS queries by type +- `hubble_dns_responses_total` — DNS responses by status +- `hubble_drop_total` — dropped packets by reason +- `hubble_tcp_flags_total` — TCP connections by flag +- `hubble_http_requests_total` — HTTP requests by method and status + +## Troubleshooting + +Check that Relay and the UI are running: + +```bash +kubectl get pods -n cozy-cilium -l k8s-app=hubble-relay +kubectl get pods -n cozy-cilium -l k8s-app=hubble-ui +``` + +Verify the metrics endpoint is serving: + +```bash +kubectl port-forward -n cozy-cilium svc/hubble-metrics 9965:9965 +curl http://localhost:9965/metrics +``` + +Confirm the scrape target exists — if the dashboards are empty but the endpoint above returns data, this is usually the missing link: + +```bash +kubectl get servicemonitor -n cozy-cilium +``` diff --git a/content/en/docs/v1.6/networking/vm-external-vlan.md b/content/en/docs/v1.6/networking/vm-external-vlan.md new file mode 100644 index 00000000..f5dcaedc --- /dev/null +++ b/content/en/docs/v1.6/networking/vm-external-vlan.md @@ -0,0 +1,130 @@ +--- +title: "Attaching a Virtual Machine to an External VLAN" +linkTitle: "VM External VLAN" +description: "Bridge a virtual machine onto a physically-routed VLAN so it shares a broadcast domain with external hardware, and why macvlan cannot work for this." +weight: 35 +--- + +This page describes how to attach a Cozystack virtual machine (the [`vm-instance`](/docs/v1.6/virtualization/vm-instance/) application) directly to an external, physically-routed VLAN — the layer-2 segment a VM needs when it must appear on the same broadcast domain as external hardware (a licensing appliance, a storage box, a gateway managed outside the cluster), with an address from that VLAN's subnet rather than from the cluster overlay. + +The default Cozystack VM networking is overlay-only (the pod network, plus optional KubeOVN [VPC subnets](/docs/v1.6/networking/vpc/)). Bridging a VM onto a real VLAN is a different pattern and has one non-obvious constraint: **it works with a Linux bridge and the `bridge` CNI plugin, and it does not work with `macvlan`.** The rest of this guide explains why and gives a working recipe. + +## Why `bridge` and not `macvlan` + +KubeVirt attaches a VM interface to a secondary network using **bridge binding** by default (this is what the `vm-instance` chart emits for every network in `.spec.networks`). With bridge binding the guest's own MAC address is placed on the wire — the launcher pod does not masquerade or translate it. + +A `macvlan` attachment is incompatible with that model. `macvlan` demultiplexes inbound frames strictly by the MAC address of the macvlan child interface. Because KubeVirt puts the *guest's* MAC on the wire — not the macvlan child's — replies from the gateway or other hosts arrive at the parent interface addressed to the guest MAC, do not match any macvlan child, and are silently dropped before they ever reach the VM. The symptom is a guest that can transmit (ARP requests and pings leave, visible in `tcpdump` on the parent interface) but never receives a reply (its neighbor entry for the gateway stays `FAILED`). As a secondary consequence, the host cannot reach macvlan children through the parent interface either, so a host-side service on the parent IP is unreachable from the VMs. + +A **Linux bridge** does not have this limitation: it forwards by learned MAC on all bridged ports, so the guest MAC is reachable, and the host can carry an address on the bridge itself to talk to the VMs. Attach the VLAN sub-interface to a bridge and point a `bridge`-type NetworkAttachmentDefinition at it. + +## Overview + +Three pieces cooperate: + +1. A **Linux bridge on each node** that enslaves the tagged VLAN sub-interface. This is node-level networking — it is configured by your node provisioning (netplan / Talos machine config / systemd-networkd), not by a Cozystack chart. +2. A **`NetworkAttachmentDefinition`** of type `bridge` referencing that bridge, created in the VM's tenant namespace. +3. The **`vm-instance`** application referencing the NetworkAttachmentDefinition by name in `.networks`, with the guest's static address supplied through cloud-init. + +## Prerequisites + +- The `multus` package is enabled (it provides the `NetworkAttachmentDefinition` CRD and the secondary-network plumbing). +- The `bridge` CNI plugin is present in `/opt/cni/bin` on every node. The `multus` package puts it there itself on every platform; see [the multus package README](https://github.com/cozystack/cozystack/blob/main/packages/system/multus/README.md) for what it stages and the opt-out, and read it before upgrading a cluster whose `/opt/cni/bin` you provision yourself. Verify with `ls /opt/cni/bin/bridge`; a missing binary makes the NetworkAttachmentDefinition fail with `failed to find plugin "bridge" in path [/opt/cni/bin]`. +- There is no IPAM plugin in this path — addresses are assigned inside the guest, not by the CNI. Plan static addresses per VM. + +## 1. Linux bridge on the node + +Create a bridge that enslaves the tagged VLAN sub-interface. The VLAN sub-interface itself carries no address; the bridge carries the host's presence on that VLAN (optional, but useful for a gateway-reachability sanity path and for any host-side service the VMs must reach). + +This example uses netplan on an Ubuntu/Debian node; the VLAN id and subnet are illustrative (`203.0.113.0/24`, VLAN 100, gateway `203.0.113.1`). Adapt to your uplink naming and to Talos or `systemd-networkd` if that is your provisioning: + +```yaml +network: + version: 2 + vlans: + # Tagged VLAN sub-interface, no address of its own — enslaved to the bridge. + uplink.100: + id: 100 + link: uplink + bridges: + br100: + interfaces: + - uplink.100 + # Optional host presence on the VLAN. Keep the node's default route on + # its management interface — do not add a default route here. + addresses: + - 203.0.113.2/24 +``` + +Notes: + +- The node's **default route must stay on the management interface.** The bridge address (if any) is only for on-VLAN reachability, not a second default gateway. +- `netplan apply` cannot move an interface into a bridge while a consumer still holds it (for example a `virt-launcher` pod using a previous `macvlan` attachment). Remove the consumer first (delete the VMI so the launcher releases the interface), then reconfigure. +- After a reboot, `systemd-networkd` may briefly report the bridge "routable" while the link is not yet actually up. If your VMs need the VLAN immediately at boot, gate their start on a reachability check, or re-run `netplan apply` until the gateway answers. + +## 2. NetworkAttachmentDefinition + +Create a `bridge`-type NetworkAttachmentDefinition in the tenant namespace that will host the VM. The `vm-instance` chart resolves a network by name **in the VM's own namespace**, so one copy must exist in every tenant namespace that runs VMs on this VLAN. + +```yaml +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: vlan100 + namespace: tenant-example +spec: + config: | + { + "cniVersion": "0.3.1", + "type": "bridge", + "bridge": "br100", + "ipam": {} + } +``` + +- `bridge` must match the bridge name from step 1 (`br100` here). +- `ipam: {}` — no cluster-side address assignment; the guest configures its address itself (step 3). + +## 3. Attach the VM and assign a static address + +Reference the NetworkAttachmentDefinition by name in the `vm-instance` values. Because the chart does not support `networkData`, the static address goes into cloud-init `userData` (`cloudInit`), written by the guest at first boot: + +```yaml +# vm-instance values +instanceType: u1.medium +instanceProfile: ubuntu +disks: + - name: example-system +networks: + - name: vlan100 +cloudInit: | + #cloud-config + write_files: + - path: /etc/netplan/60-vlan100.yaml + permissions: "0600" + content: | + network: + version: 2 + ethernets: + # Match the second NIC (the pod-network NIC is the first). Use the + # interface that comes up without a DHCP lease. + enp2s0: + addresses: + - 203.0.113.10/24 + runcmd: + - netplan apply +``` + +The VM ends up with two interfaces: the always-present **pod-network** NIC (`default`, used for cluster-internal traffic and for the `vm-instance` external-access features) and the **VLAN** NIC. The `/24` address above brings up only the connected route for the VLAN subnet — it adds no default route, so the guest's egress stays wherever you want it (typically the pod NIC). If the VLAN is meant to be the guest's default gateway instead, add a default route under the VLAN NIC and remove it from the pod NIC. + +## Gotchas + +- **VMs are dual-homed.** The `vm-instance` chart always adds the pod-network NIC in addition to any `networks` you declare; there is no single-homed (VLAN-only) option today. Address the VLAN NIC inside the guest and leave the pod NIC to the cluster. +- **No `networkData`.** The chart wires cloud-init through `userData` only, so in-guest static configuration (netplan `write_files` plus `netplan apply`, as above) is the way to assign the VLAN address. +- **MAC changes on VM re-creation.** KubeVirt generates a fresh guest MAC each time the VM object is re-created, and `vm-instance` exposes no way to pin it, so re-creating a VM changes its MAC. The upstream gateway then holds a stale ARP entry for the old MAC for a few minutes, so "gateway unreachable" immediately after re-creating a VM is expected — wait for the ARP entry to age out (roughly five minutes) rather than treating it as a fault. +- **Host-to-VM traffic.** If the host must talk to the VMs (a proxy, a health check), give the bridge a host address on the VLAN (step 1) — traffic through a bare VLAN sub-interface to bridge-attached guests will not work the way `macvlan` users expect. + +## See also + +- [Attaching GPUs to virtual machines](/docs/v1.6/virtualization/gpu/) — passing NVIDIA GPUs and vGPU profiles into the same VMs. +- [Networking architecture](/docs/v1.6/networking/architecture/) — how the default overlay data plane is put together. +- KubeVirt user guide, [Interfaces and Networks](https://kubevirt.io/user-guide/network/interfaces_and_networks/) — bridge binding versus other binding methods. diff --git a/content/en/docs/v1.6/operations/troubleshooting/gpu-operator-host-driver.md b/content/en/docs/v1.6/operations/troubleshooting/gpu-operator-host-driver.md new file mode 100644 index 00000000..5e88a6ab --- /dev/null +++ b/content/en/docs/v1.6/operations/troubleshooting/gpu-operator-host-driver.md @@ -0,0 +1,147 @@ +--- +title: "GPU Passthrough Fails on a Host with a Pre-installed NVIDIA Driver" +linkTitle: "GPU Operator: host driver" +description: "Why vfio-manager refuses to bind vfio-pci when the node already has an apt-installed NVIDIA driver, and how to recover the node." +weight: 30 +--- + +The `default` (passthrough) variant of the `cozystack.gpu-operator` package assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**. On a node that already has the NVIDIA driver installed through the distro package manager, the operator detects it, declines to touch it, and the passthrough setup never completes. + +Verified 2026-05-28 against the `gpu-operator` chart v26.3.1 with `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`, on an Ubuntu 24.04 host carrying `nvidia-driver-580-open` 580.82.07. + +{{< note >}} + +If you want GPUs in **containers** rather than in VMs, you do not need this recovery at all — use the [`container` variant](/docs/v1.6/operations/gpu-container-workloads/), which is designed for exactly this host shape and keeps the host driver in place. + +{{< /note >}} + +## Symptom + +`kubectl get pods -n cozy-gpu-operator` shows `nvidia-vfio-manager-*` stuck in `Init:Error` or `Init:CrashLoopBackOff` — the init container exits non-zero after detecting the host driver, so kubelet keeps restarting it. + +Its log shows it skipped the bind step: + +```text +Host driver detected: 580.82.07 +NVIDIA GPU driver is already pre-installed on the node, + disabling the containerized driver +Labeling node with nvidia.com/gpu.deploy.driver=pre-installed +``` + +`nvidia-sandbox-validator` then crashloops: + +```text +Error: error validating vfio-pci driver installation: + device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia' +``` + +`lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and this reports `{}` — no GPU resource was registered: + +```bash +kubectl get node -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))' +``` + +## Why it happens + +The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand. That path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present". File existence is not pre-tested — if `nvidia-smi` is missing the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed. + +On a positive detection the binary logs `Host driver detected: `, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, and exits. + +**`FORCE_REINSTALL` does not bypass this.** `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env and flag pair, but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top. Operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself. + +## Recovery: clean the host + +Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. + +Two pitfalls to avoid. `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, so a blanket autoremove can take RDMA out on a converged GPU plus RDMA host. And a hardcoded `apt purge 'nvidia-*' 'cuda-*'` pattern list is fragile — `apt` treats `*` as a cache-wide regex and **aborts the entire transaction, purging nothing**, if any pattern matches nothing in the cache (for example `cuda-*` on a host without NVIDIA's CUDA repo). Build the list from what is actually installed instead: + +```bash +# dpkg-query patterns are true globs over INSTALLED packages: no +# zero-match abort (unlike apt's cache-wide regex) and no accidental +# substring over-match. List first, then review before purging. +dpkg-query -W -f '${Package}\n' 'nvidia-*' 'libnvidia-*' 'cuda-*' 2>/dev/null +``` + +Review the list before purging: + +- `nvidia-dkms-*` and `nvidia-kernel-*` are the load-bearing kernel pieces — without removing them DKMS rebuilds `nvidia.ko` on the next reboot and the blacklist below is bypassed by any explicit `modprobe`. +- On a **converged GPU plus RDMA host**, drop any `libnvidia-*` that belong to NVIDIA DOCA / Mellanox OFED — purging those breaks RDMA. +- If the list is **empty**, the driver was installed with NVIDIA's `.run` installer rather than apt — run `sudo nvidia-uninstall` instead of the purge below. + +Then purge the reviewed list, blacklist the modules, and rebuild the initramfs: + +```bash +# Replace with the packages you kept from the list above. +sudo apt purge nvidia-driver-580-open nvidia-dkms-580-open <...> + +sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF' +blacklist nouveau +blacklist nvidia +blacklist nvidia_drm +blacklist nvidia_modeset +blacklist nvidia_uvm +blacklist nvidia_peermem +EOF + +# -k all rebuilds EVERY installed kernel's initramfs (plain -u touches +# only the running kernel), so a just-upgraded kernel also boots with +# the blacklist and cannot re-claim the GPU. +sudo update-initramfs -u -k all +sudo reboot +``` + +## Confirm the host is clean + +The init container exits non-zero after detecting a host driver, so both DaemonSet pods sit in `Init:CrashLoopBackOff` and **will retry on their own** — deleting them just skips the up-to-five-minute backoff window. + +```bash +# Should print nothing. +lsmod | grep -E '^(nvidia|nouveau)' + +# command -v matches what isHostDriver does (PATH lookup inside the +# chroot), so it also catches /usr/local/bin/nvidia-smi left by a .run +# or CUDA-toolkit install. Prints "ok: gone" on success. +command -v nvidia-smi >/dev/null && echo "STILL PRESENT — purge incomplete" || echo "ok: gone" + +# A leftover DKMS module rebuilds nvidia.ko on the next kernel update +# and an explicit modprobe bypasses the blacklist — should print nothing. +dkms status | grep -i nvidia + +# Skip the CrashLoopBackOff backoff window by deleting the stuck pods. +# The DaemonSet labels are operator-managed and not stable across +# gpu-operator versions, so delete by name pattern — anchored so the +# match is the two DaemonSets and nothing else. +kubectl -n cozy-gpu-operator get pods -o name \ + | grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \ + | xargs -r kubectl -n cozy-gpu-operator delete +``` + +Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource: + +```bash +lspci -nnk -d 10de: | grep 'Kernel driver in use' +# Kernel driver in use: vfio-pci + +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com +``` + +## Clear the leftover label + +One label is **not** cleaned up automatically. The init container set `nvidia.com/gpu.deploy.driver=pre-installed`, and the operator's success path restores only the operand labels — `rescheduleGPUOperatorComponents` in `k8s-driver-manager` v0.10.0 touches the validator, toolkit and device-plugin labels, not `deploy.driver`. So the node keeps `pre-installed` indefinitely: it is the same label the Symptom section uses as evidence, and it would disable the containerized-driver DaemonSet if the node later switches to a container workload. + +```bash +# Nothing on the success path resets this label; clear it so it does not +# mislead future debugging or block a later container-workload switch. +kubectl label node nvidia.com/gpu.deploy.driver- +``` + +## Known limitation + +The skip-on-pre-installed behaviour lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. + +Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads therefore cannot share the same GPU with the passthrough variant. Two paths out: + +- **Use the [`container` variant](/docs/v1.6/operations/gpu-container-workloads/)** if the workloads can be containers rather than VMs. It targets the apt-installed-driver host shape and exposes GPUs to pods without unbinding the host driver, so no purge is needed. +- **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191), still open. + +Talos is unaffected: the Talos image ships only the `vfio-pci` extension and no host NVIDIA stack, so the clean-host check passes trivially. This page applies to distributions where you installed the host driver yourself — typically Ubuntu, Debian or RHEL with `apt install nvidia-driver-*` or the equivalent. diff --git a/content/en/docs/v1.6/virtualization/gpu.md b/content/en/docs/v1.6/virtualization/gpu.md index f4da0f67..4e2da976 100644 --- a/content/en/docs/v1.6/virtualization/gpu.md +++ b/content/en/docs/v1.6/virtualization/gpu.md @@ -270,7 +270,7 @@ GPU passthrough assigns an entire physical GPU to a single VM. To share one GPU ### vGPU (Virtual GPU) -NVIDIA vGPU uses mediated devices (mdev) to create virtual GPUs assignable to VMs. This is the only production-ready solution for GPU sharing between VMs. +NVIDIA vGPU is the only production-ready solution for GPU sharing between VMs. Two host-side models exist depending on GPU generation: mediated devices (mdev) on Pascal through Ampere, and SR-IOV virtual functions on Ada Lovelace and newer. See [NVIDIA vGPU for virtual machines](/docs/v1.6/virtualization/vgpu/) for the full setup, including profile assignment and DLS licensing. **Requirements:** - NVIDIA vGPU license (commercial, purchased from NVIDIA) diff --git a/content/en/docs/v1.6/virtualization/vgpu.md b/content/en/docs/v1.6/virtualization/vgpu.md new file mode 100644 index 00000000..2193b5aa --- /dev/null +++ b/content/en/docs/v1.6/virtualization/vgpu.md @@ -0,0 +1,292 @@ +--- +title: "NVIDIA vGPU for Virtual Machines" +linkTitle: "vGPU" +description: "Slice one physical NVIDIA GPU across several virtual machines with the vgpu variant of the GPU Operator package, including SR-IOV profile assignment and DLS licensing." +weight: 45 +--- + +This page describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines. For handing a whole GPU to one VM, see [GPU passthrough](/docs/v1.6/virtualization/gpu/); for GPUs in containers rather than VMs, see [containerized GPU workloads](/docs/v1.6/operations/gpu-container-workloads/). + +Verified 2026-04-29 against KubeVirt `main` (`virt-handler` nightly `20260429_74d7c52588`), the `vgpu` variant of `cozystack.gpu-operator`, the NVIDIA vGPU 20.0 host driver `595.58.02` and GRID guest driver `595.58.03`. + +## Two driver models + +NVIDIA's vGPU driver uses two different host-side models depending on GPU generation: + +- **Mediated devices (mdev)** — Pascal / Volta / Turing / Ampere up to A100 and A30. The driver creates `mdev` parent devices under `/sys/class/mdev_bus/`; KubeVirt advertises them via `permittedHostDevices.mediatedDevices`. +- **SR-IOV with per-VF sysfs** — Ada Lovelace (L4, L40, L40S, …) and Blackwell (B100, …) on the vGPU 17/20 driver branch. The driver creates SR-IOV virtual functions; profile selection happens via `/sys/bus/pci/devices//nvidia/current_vgpu_type`. KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` after [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890). + +This guide focuses on the **SR-IOV path**, which is the only model NVIDIA supports for current data-centre GPUs. Mdev is mentioned for completeness; for Pascal to Ampere refer to the upstream NVIDIA GPU Operator documentation. + +## Prerequisites + +- An Ada Lovelace or newer NVIDIA GPU that supports SR-IOV vGPU (L4, L40, L40S, and similar). +- Ubuntu 24.04 host OS. Older Ubuntu releases also work if the upstream `gpu-driver-container` repository has a matching `vgpu-manager/` Dockerfile. **Talos Linux is not recommended** for vGPU: NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero [closed siderolabs/extensions#461](https://github.com/siderolabs/extensions/issues/461) noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that includes the driver in-tree is therefore not feasible without a private fork that violates the EULA. +- KubeVirt with [kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890) ("vGPU: SRIOV support", merged to `main` 2026-04-10). Targeted at the next minor release (v1.9.0); track the pull request for the actual release tag. Released tags up to and including v1.8.x do not include the patch and backports are not planned. If you need vGPU before v1.9.0 lands you have to run a `main`-based nightly build of `virt-handler`; the rest of the operator can stay on the latest released tag. +- An NVIDIA vGPU Software or NVIDIA AI Enterprise subscription (the `.run` is not redistributable). +- A reachable NVIDIA Delegated License Service (DLS) instance and a matching `client_configuration_token.tok` file. + +## Variants + +The `gpu-operator` package exposes three variants. This page is vGPU-focused; the variant inventory is shared. + +- **`default`** — passthrough mode (`vfio-pci`). The whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, so no proprietary driver is needed on the host. On a host that already carries an apt-installed NVIDIA driver this variant will not complete — see [GPU passthrough fails on a host with a pre-installed NVIDIA driver](/docs/v1.6/operations/troubleshooting/gpu-operator-host-driver/). +- **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU. +- **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin, on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit`. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. See [containerized GPU workloads](/docs/v1.6/operations/gpu-container-workloads/). + +## Building the vGPU Manager image + +The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager//` and is the source of truth for build arguments, base images and supported OS releases. Follow the README in that repository. + +The proprietary `.run` is the **Linux KVM** variant, not the Ubuntu KVM `.deb` (which ships pre-built modules for stock kernels only). It comes from the [NVIDIA Licensing Portal](https://ui.licensing.nvidia.com) under an NVIDIA AI Enterprise or vGPU subscription. + +{{< warning >}} + +**EULA:** never push the resulting image to a publicly readable registry. Use a private registry — an in-cluster Harbor works well as a non-proxy project. + +{{< /warning >}} + +## Deploying with the vgpu variant + +The platform's `iaas` bundle deploys the gpu-operator Package CR when `cozystack.gpu-operator` is in `bundles.enabledPackages` and `bundles.iaas.gpuOperatorVariant: vgpu` is set. The vGPU Manager image is proprietary and not redistributable, so the bundle does not ship a default tag — build the container per the upstream `gpu-driver-container` recipe and supply the private-registry coordinates through platform values: + +```yaml +bundles: + iaas: + enabled: true + gpuOperatorVariant: vgpu + enabledPackages: + - cozystack.gpu-operator + +gpu: + vgpuManager: + repository: registry.example.com/nvidia + image: vgpu-manager + version: "595.58.02-ubuntu24.04" + # imagePullSecrets lives per-component (vgpuManager, driver, + # validator, dcgmExporter, …). The value is a list of strings, + # not [{name: ...}]. + imagePullSecrets: + - nvidia-registry-secret +``` + +The platform forwards `gpu.vgpuManager` into the emitted gpu-operator Package CR's `components.gpu-operator.values.gpu-operator.vgpuManager`, so the bundle handles the variant and image coordinates in one place. If you need to override anything else on the gpu-operator chart (driver, validator, dcgmExporter, custom node selectors), hand-craft a `Package` CR named `cozystack.gpu-operator` with the full `components.gpu-operator.values` block — that takes precedence over the bundle render. + +The `nvidia-registry-secret` should be a docker-registry Secret created beforehand in `cozy-gpu-operator`. + +Verify the DaemonSet is running and `nvidia.ko` loads on every GPU node: + +```bash +kubectl -n cozy-gpu-operator get pods -l app=nvidia-vgpu-manager-daemonset +kubectl -n cozy-gpu-operator exec -it -- nvidia-smi +``` + +`nvidia-smi` should enumerate the physical GPUs and report `Host VGPU Mode : SR-IOV`. + +## Profile assignment (SR-IOV path) + +{{< caution >}} + +**The `vgpu` variant is experimental on Ada and newer, and ships without a profile-assignment loop.** NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada and newer — the DaemonSet errors with "no parent devices found for GPU at index '0'" and is therefore disabled by default in `values-vgpu.yaml`. Until an SR-IOV-aware controller ships, profile assignment is an out-of-band step that must be re-applied after every node reboot (`current_vgpu_type` resets to 0 on PCIe re-enumeration). Without this step `permittedHostDevices.pciHostDevices` reports zero allocatable resources and no VM can request the vGPU. **Do not deploy the `vgpu` variant in production until you have an automated profile-assignment mechanism in place** — typically a small DaemonSet that reads a ConfigMap (` = `) and writes the corresponding `current_vgpu_type` files at boot. + +{{< /caution >}} + +Once `nvidia.ko` is loaded the driver enables SR-IOV (16 VFs per L40S by default). Each VF needs a vGPU profile written to its sysfs: + +```bash +# from inside the nvidia-vgpu-manager-daemonset pod (privileged, hostPID) +echo 1155 > /sys/bus/pci/devices/0000:02:00.5/nvidia/current_vgpu_type +``` + +The numeric profile ID can be discovered per-VF: + +```bash +cat /sys/bus/pci/devices/0000:02:00.5/nvidia/creatable_vgpu_types +``` + +For Pascal to Ampere GPUs (V100, T4, A100, A30) the mdev model still applies. Flip `vgpuDeviceManager.enabled: true` in your Package CR overrides — NVIDIA's device manager works correctly there. + +## KubeVirt configuration + +When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step. + +If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly — typically to apply overrides the bundle does not expose — the platform does **not** auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow you also hand-craft a `cozystack.kubevirt` Package CR with `components.kubevirt.values.extraFeatureGates: [HostDevices]` and the appropriate `permittedHostDevices` block. The escape-hatch values shape under `.gpu` below is documented for the bundle-managed flow only; the manual Package-CR override path takes precedence over the bundle render whenever both exist. + +- `developerConfiguration.featureGates` gets `HostDevices` appended (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `spec.template.spec.domain.devices.hostDevices` without it). +- `permittedHostDevices.pciHostDevices` is filled from `packages/core/platform/files/gpu-passthrough-defaults.yaml` in the [cozystack repository](https://github.com/cozystack/cozystack) when `bundles.iaas.gpuOperatorVariant: default` (the package default). The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and Volta (V100/V100S). All entries carry `externalResourceProvider: true` because the resource names come from `nvidia-sandbox-device-plugin`, not from KubeVirt's in-tree device plugin. +- `permittedHostDevices.mediatedDevices` is filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu`. This list only *exposes*, by profile name (`mdevNameSelector`), mdevs that the GPU Operator's vGPU Device Manager *creates* on the node; the platform does not ship a numeric `mediatedDevicesConfiguration` default (those `nvidia-NNN` type ids are per-SKU and per-driver sysfs indices with no portable value — set `.gpu.mediatedDevicesConfiguration` yourself, with host-verified ids, only if you want KubeVirt rather than the Device Manager to create mdevs). The starter set covers Pascal to Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and Blackwell SR-IOV vGPU are out of scope for the chart's default list; advertise those VFs via the user-override hook below. + +### Extending or replacing the default table + +The platform exposes three knobs under `.gpu`: + +```yaml +gpu: + # Extend the platform defaults with cluster-specific entries. Both list + # keys are read in both variants: pciHostDevices feeds the passthrough + # (vfio-pci) path AND the post-kubevirt#16890 SR-IOV vGPU VF path on + # Ada Lovelace / Blackwell; mediatedDevices feeds the pre-#16890 mdev + # path on Pascal–Ampere. Both render into the same KubeVirt CR. + permittedHostDevices: + pciHostDevices: + - pciVendorSelector: "10DE:26B9" # L40S, advertised as a VF for SR-IOV vGPU + resourceName: nvidia.com/L40S-24Q + # externalResourceProvider is intentionally omitted here: after + # kubevirt/kubevirt#16890, virt-handler's in-tree device plugin + # advertises the resource directly, no sandbox plugin in the loop. + mediatedDevices: [] + # mediatedDevicesConfiguration makes KubeVirt itself create mdevs (vgpu + # mode). No platform default: mdev creation is normally delegated to the + # vGPU Device Manager (name-based), and these mediatedDeviceTypes are + # host/driver-specific nvidia-NNN sysfs indices (look yours up via + # /sys/bus/pci/devices//mdev_supported_types/*/name). Set this only + # to opt into KubeVirt-driven creation; mergeOverwrite REPLACES a + # supplied top-level key wholesale. + mediatedDevicesConfiguration: {} + # Wipe the platform defaults entirely and ship only the cluster's + # curated lists. Useful for non-NVIDIA-only clusters and strict + # allowlist requirements. + replaceDefaults: false +``` + +`replaceDefaults: false` (the default) appends user entries to the NVIDIA defaults. `replaceDefaults: true` drops the NVIDIA table entirely — if you do not then supply your own `pciHostDevices` or `mediatedDevices` list, the rendered KubeVirt CR has no `permittedHostDevices` block and the admission webhook rejects every GPU VM. + +### Resource names from `nvidia-sandbox-device-plugin` + +The `resourceName` strings in `gpu-passthrough-defaults.yaml` are what `nvidia-sandbox-device-plugin` (`nvcr.io/nvidia/kubevirt-gpu-device-plugin`) advertises: it derives each slug mechanically from the device's PCI-IDs database name by uppercasing it, turning `/`, `.` and whitespace into `_`, and stripping the remaining non-alphanumerics (the `[` and `]`). So `TU104GL [Tesla T4]` becomes `nvidia.com/TU104GL_TESLA_T4` and `GA100GL [A30 PCIe]` becomes `nvidia.com/GA100GL_A30_PCIE` — the slug carries every token the PCI-IDs string holds (the `GL` die suffix, the `Tesla` brand on Turing and Volta, form factor, memory), not a tidy `_`. The names track the pci.ids snapshot bundled in the plugin image, so a different plugin build can publish a different string — check with `kubectl describe node | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor and device IDs themselves are stable across driver versions. + +### SR-IOV PF versus VF on Ada Lovelace and newer + +On L40S and other Ada Lovelace cards the SR-IOV VFs report the same PCI device ID as the PF — `lspci -nn -d 10de:` on the host shows both as `[10de:26b9]`. `virt-handler` distinguishes them by "is a VF and has a vGPU profile", so a single `pciVendorSelector` matches the right set. Verify on your specific GPU before assuming this — some other generations split PF and VF IDs. + +`externalResourceProvider: true` is **not** required when the resource is advertised by `virt-handler`'s in-tree device plugin (the SR-IOV path after kubevirt#16890). The platform passthrough defaults include the flag because that path is driven by the external sandbox plugin. + +### Verifying allocatable capacity + +```bash +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable.nvidia\.com/L40S-24Q}{"\n"}{end}' +``` + +## Licensing (DLS) + +vGPU 17/20 uses the NVIDIA Delegated License Service. The legacy `ServerAddress=` and `ServerPort=7070` lines in `gridd.conf` are no longer authoritative — `nvidia-gridd`, running **inside the guest**, reads the DLS endpoint from the ClientConfigToken file directly. + +The host vGPU Manager DaemonSet does not request a license — it only enables SR-IOV and loads `nvidia.ko`. Licensing is consumed entirely by the guest. The gpu-operator chart's `driver.licensingConfig.secretName` would mount the Secret into the **driver pod on the host**, where it has no effect for SR-IOV vGPU; do not wire the licensing Secret through it. + +Instead, deliver the token and `gridd.conf` to the guest via cloud-init or a containerDisk overlay: + +```yaml +# inside the VirtualMachine cloudInitNoCloud userData +write_files: +- path: /etc/nvidia/ClientConfigToken/client_configuration_token.tok + # 0744 follows NVIDIA's recommendation in the Virtual GPU Software + # Licensing User Guide ("Configuring a Licensed Client on Linux"): + # nvidia-gridd does not necessarily run as the file owner. + # https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/ + permissions: '0744' + encoding: b64 + content: +- path: /etc/nvidia/gridd.conf + permissions: '0644' + content: | + # FeatureType selects which vGPU Software license the guest requests. + # 0 — unlicensed state (no license requested; Q profiles run in + # reduced mode after the grace period). + # 1 — NVIDIA vGPU. The driver auto-selects the correct license + # type from the configured vGPU profile (Q → vWS, B → vPC, + # A → vCS / Compute). Use this for SR-IOV vGPU profiles. + # 2 — explicitly NVIDIA RTX Virtual Workstation. + # 4 — explicitly NVIDIA Virtual Compute Server. + FeatureType=1 +``` + +Verify activation inside the guest: + +```bash +nvidia-smi -q | grep 'License Status' +# License Status : Licensed +``` + +If the guest reports `Unlicensed (Unrestricted)` for more than a couple of minutes, check `journalctl _COMM=nvidia-gridd` for handshake errors against the DLS endpoint baked into the token. + +### Migrating from chart v25.x + +Upstream deprecated `driver.licensingConfig.configMapName` in favour of `driver.licensingConfig.secretName`. The old key still works but emits a deprecation warning at render time. If your existing `Package` CR set the licensing reference via `configMapName`, switch it to `secretName` on this upgrade — the Secret content (`gridd.conf` and the ClientConfigToken) does not need to change. This applies to passthrough deployments that drove host-side licensing through the gpu-operator chart; SR-IOV vGPU does not consume the host-side licensing knob at all, as above. + +## Sample VirtualMachine + +Either `hostDevices` or `gpus` accepts the resource (the upstream KubeVirt API resolves both PCI and mediated-device pools), but the convention is to use `hostDevices` for VF-style PCI passthrough: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: vgpu-smoke + namespace: tenant-example +spec: + runStrategy: Always + template: + spec: + domain: + cpu: + cores: 4 + memory: + guest: 8Gi + devices: + disks: + - name: rootdisk + disk: + bus: virtio + interfaces: + - name: default + masquerade: {} + hostDevices: + - name: gpu0 + deviceName: nvidia.com/L40S-24Q + networks: + - name: default + pod: {} + volumes: + - name: rootdisk + # A 2.4 GiB containerDisk overlay is too small to install + # the GRID guest driver in-place. Use a CDI DataVolume of + # 20 GiB+ in production. + containerDisk: + image: quay.io/containerdisks/ubuntu:24.04 +``` + +Inside the guest, install the GRID driver from the `.run` — the GUEST `.run`, distinct from the host `vgpu-kvm` package — after which `nvidia-smi` should report the configured profile: + +```text +| 0 NVIDIA L40S-24Q Off | 00000000:0E:00.0 Off | 0 | +| 17 MiB / 24576 MiB P0 Default | +``` + +## Profile reference (L40S) + +L40S supports the full Q (RTX vWS), B (vPC) and A (vCS / Compute) profile families. The numeric IDs come from the driver and are visible in `creatable_vgpu_types`: + +| Profile | Frame Buffer | Max instances per L40S | Use case | +| --- | --- | --- | --- | +| L40S-1Q | 1 GB | 48 | Light 3D / VDI | +| L40S-2Q | 2 GB | 24 | Medium 3D / VDI | +| L40S-4Q | 4 GB | 12 | Heavy 3D / VDI | +| L40S-6Q | 6 GB | 8 | Professional 3D | +| L40S-8Q | 8 GB | 6 | AI / ML inference | +| L40S-12Q | 12 GB | 4 | AI / ML training | +| L40S-24Q | 24 GB | 2 | Large AI workloads | +| L40S-48Q | 48 GB | 1 | Full GPU equivalent | + +Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Documentation](https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/). + +## OS support summary + +The `container` column assumes the host already ships the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager, with the `nvidia` runtime registered in containerd. With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override. Talos installs the driver under a non-standard prefix, so the operator does not find it at the default location — see `packages/system/gpu-operator/examples/` in the [cozystack repository](https://github.com/cozystack/cozystack) for the Talos-specific path with a compat DaemonSet and an explicit `hostPaths.driverInstallDir` override. + +| Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) | +| --- | --- | --- | --- | +| Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see [host-driver recovery](/docs/v1.6/operations/troubleshooting/gpu-operator-host-driver/) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver plus nvidia-container-toolkit | +| Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | +| Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | +| Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus an `nvidia-driver` patch for usr-merge (details pending) | ⚠️ same patch plus own Dockerfile fork | ✅ | +| Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run` | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` as a starting point |