[KVM + Ceph] Snapshots/Templates saved locally (/snapshots, /template) instead of NFS secondary storage

[jeanvetorello]

problem

Actual behavior

2025-09-22 19:24:58,440 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:03909a05) Attempting to backup RBD snapshot cloudstack/07....
2025-09-22 19:24:58,441 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:03909a05) Attempting to create /snapshots/2/505 recursively for snapshot storage
2025-09-22 19:24:58,449 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:03909a05) Backing up RBD snapshot cloudstack/07a70846-dd72-4d7e-8b38-96eccd2458ef@bbda9153-f224-47c3-95ac-f368d849da20 to /snapshots/2/505/bbda9153-f224-47c3-95ac-f368d849da20

The KVM agent writes to local paths:

/snapshots///
/template/tmpl//

These directories exist only locally on the host, not on the shared NFS.

As a result, SSVM cannot find the file and the job fails with:

cp: -r not specified; omitting directory '/mnt/SecStorage/.../snapshots/.../'
java.lang.NullPointerException: Cannot read field "format" because "newInfo" is null

Logs

From KVM Agent (/var/log/cloudstack/agent/agent.log):

Backing up RBD snapshot cloudstack/@ to /snapshots/2/505/
Executing command [qemu-img convert -O qcow2 -U rbd:cloudstack/... /snapshots/2/505/].

From SSVM (/var/log/cloud.log):

cp .../snapshots/.../ .../template/.../.raw
cp: -r not specified; omitting directory '/mnt/SecStorage/.../snapshots/.../'
java.lang.NullPointerException: Cannot read field "format" because "newInfo" is null

Workaround

Mount the NFS secondary storage explicitly on the KVM host at /snapshots and /template:

192.168.1.80:/cloudstack/template /template nfs defaults,_netdev 0 0
192.168.1.80:/cloudstack/snapshots /snapshots nfs defaults,_netdev 0 0

After this mount, qemu-img convert outputs go directly to the NFS and templates work correctly.

versions

Environment

CloudStack version: 4.21.0.0

Hypervisor: KVM (Ubuntu 22.04)

Primary storage: Ceph RBD (poolType=RBD)

Secondary storage: NFS (nfs://192.168.1.80/cloudstack)

Agents: qemu-utils, ceph-common, nbdkit installed

The steps to reproduce the bug

Steps to reproduce

Deploy CloudStack with:

KVM host using Ceph RBD as primary storage

NFS secondary storage

Create a VM and take a snapshot of the ROOT volume.

Try to create a template from this snapshot.
...

What to do about it?

No response

[kiranchavala]

@jeanvetorello could not reproduce the issue

Could you send the logs to investigate further

1. SSVM logs ( /var/log/cloud.log)
2. Cloudstack management server logs (/var/log/cloudstack/management/management-server.log)
3. KVM agent logs (/var/log/cloudstack/agent/agent.log)

What is ceph version you are using

[jeanvetorello]

The qemu-img convert -O qcow2 -U command executed on the KVM host wrote the snapshot output into the local /snapshots/... path, which in practice was mapped to the NFS share accessible by the SSVM.

This made the SSVM able to see
I am not sure if this is the expected behavior, or if it only worked because of the workaround I applied. Does it make sense that the conversion should always target the NFS path so the SSVM can access it?

agent.log

kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:03909a05) Successfully refreshed pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 Capacity: (500.00 GB) 536866717696 Used: (37.47 GB) 40234788865 Available: (437.62 GB) 469891428352 2025-09-22 19:24:58,440 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:03909a05) Attempting to backup RBD snapshot cloudstack/07a70846-dd72-4d7e-8b38-96eccd2458ef@bbda9153-f224-47c3-95ac-f368d849da20 2025-09-22 19:24:58,441 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:03909a05) Attempting to create /snapshots/2/505 recursively for snapshot storage 2025-09-22 19:24:58,449 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:03909a05) Backing up RBD snapshot cloudstack/07a70846-dd72-4d7e-8b38-96eccd2458ef@bbda9153-f224-47c3-95ac-f368d849da20 to /snapshots/2/505/bbda9153-f224-47c3-95ac-f368d849da20 2025-09-22 19:24:58,449 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:03909a05) Looking for libvirtd connection at: qemu:///system 2025-09-22 19:24:58,451 DEBUG [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:03909a05) Executing command [qemu-img convert -O qcow2 -U rbd:cloudstack/07a70846-dd72-4d7e-8b38-96eccd2458ef@bbda9153-f224-47c3-95ac-f368d849da20:mon_host=192.168.27.80:auth_supported=cephx:id=cloudstack:key=a/v9hLKWgUUIEgig==:rbd_default_format=2:client_mount_timeout=30 /snapshots/2/505/bbda9153-f224-47c3-95ac-f368d849da20 ]. 2025-09-22 19:25:00,261 DEBUG [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:03909a05) Successfully executed process [15739] for command [qemu-img convert -O qcow2 -U rbd:cloudstack/07a70846-dd72-4d7e-8b38-96eccd2458ef@bbda9153-f224-47c3-95ac-f368d849da20:mon_host=192.168.27.80:auth_supported=cephx:id=cloudstack:key=a/v9hLKWgUUIEgig==:rbd_default_format=2:client_mount_timeout=30 /snapshots/2/505/bbda9153-f224-47c3-95ac-f368d849da20 ]. 2025-09-22 19:25:00,262 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:03909a05) Finished backing up RBD snapshot cloudstack/07a70846-dd72-4d7e-8b38-96eccd2458ef@bbda9153-f224-47c3-95ac-f368d849da20 to /snapshots/2/505/bbda9153-f224-47c3-95ac-f368d849da20 Snapshot size: (1.56 GB) 1678704640 2025-09-22 19:25:00,263 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:03909a05) Attempting to remove storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 from libvirt 2025-09-22 19:25:00,263 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:03909a05) Looking for libvirtd connection at: qemu:///system 2025-09-22 19:25:00,266 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:03909a05) Storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 was successfully removed from libvirt. 2025-09-22 19:25:00,267 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:03909a05) Seq 40021-4170051779968368652: { Ans: , MgmtId: 206863102531824, via: 40021, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/505/bbda9153-f224-47c3-95ac-f368d849da20","kvmIncrementalSnapshot":"false","id":"0","quiescevm":"false","physicalSize":"1678704640","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }

i didn't find the correct id but is the same.

==> /var/log/cloud.log <== 2025-09-22T19:09:18,947 WARN [utils.script.Script] (AgentRequest-Handler-3:[]) Process [5235] for command [/bin/bash -c cp /mnt/SecStorage/63b29c8e-e596-3f88-a000-d6cc15ab94e5/snapshots/2/505/aa47846d-1c71-4104-82ed-4f775e9d39f8 /mnt/SecStorage/63b29c8e-e596-3f88-a000-d6cc15ab94e5/template/tmpl/2/256/aa47846d-1c71-4104-82ed-4f775e9d39f8.raw ] encountered the error: [cp: -r not specified; omitting directory '/mnt/SecStorage/63b29c8e-e596-3f88-a000-d6cc15ab94e5/snapshots/2/505/aa47846d-1c71-4104-82ed-4f775e9d39f8'].

Regarding Ceph, everything seems fine — snapshots are created and downloaded without issues. I believe the problem is more related to how CloudStack declares or handles the paths.

@jeanvetorello could not reproduce the issue

Could you send the logs to investigate further

1. SSVM logs ( /var/log/cloud.log)
2. Cloudstack management server logs (/var/log/cloudstack/management/management-server.log)
3. KVM agent logs (/var/log/cloudstack/agent/agent.log)

What is ceph version you are using

[kiranchavala]

@jeanvetorello thanks for the details

Let me try with ubuntu 22.04 kvm host and check if the issue is reproducible

[jeanvetorello]

This issue was caused by an adjustment I’m already handling in another ticket. It is a duplicate. Sorry for the confusion.