Local Storage migration fails due to storage pool not found on destination KVM host

[yadvr]

On a 4.18.1.0 RC2 adv zone KVM env using local only storage and local storage for systemvms, migration fails from kvm1 to kvm2 due to the following seen in the target kvm host

2023-09-05 21:12:06,645 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:f38dafe4) Attempting to create volume 23ef733c-c26e-405a-829e-6e0010b3cbeb (Filesystem) in pool 3a11d1ea-eff4-4f5b-9b7c-e0ff89a8c9dd with size (2.20 GB) 2361393152
2023-09-05 21:12:06,691 INFO  [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:f38dafe4) No existing libvirtd connection found. Opening a new one
2023-09-05 21:12:06,692 WARN  [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:f38dafe4) Can not find a connection for Instance i-2-453-VM. Assuming the default connection.
2023-09-05 21:12:08,499 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:f38dafe4) Trying to fetch storage pool 8aa9768c-cbcf-4e8e-8875-f94a7f9445b6 from libvirt
2023-09-05 21:12:08,500 WARN  [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:f38dafe4) Caught:
com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 8aa9768c-cbcf-4e8e-8875-f94a7f9445b6 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '8aa9768c-cbcf-4e8e-8875-f94a7f9445b6'
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:278)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:264)

The above can sometimes cause the VM to crash as it fails right at the last step on the dest. kvm host end, and CloudStack may further report that domain is not running.

However, systemvm migration with storage work. I'm not sure if this is an env issue or a bug.

ISSUE TYPE

• Bug Report
• Improvement Request
• Enhancement Request
• Feature Idea
• Documentation Report
• Other

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[weizhouapache]

@rohityadavcloud
Thanks for raising the issue.
I will try to reproduce the issue

[weizhouapache]

I got an error when migrate the default template

and management-server log shows
2023-09-05 16:47:24,215 DEBUG [c.c.a.t.Request] (AgentManager-Handler-18:null) (logid:) Seq 2-4185251428710744296: Processing: { Ans: , MgmtId: 167780532, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateAnswer":{"result":"false","details":"Exception during migrate: org.libvirt.LibvirtException: operation failed: migration of disk vda failed: Input/output error","wait":"0","bypassHostMaintenance":"false"}}] }

and then

2023-09-05 16:47:24,220 DEBUG [c.c.a.t.Request] (Work-Job-Executor-4:ctx-9389653d job-33/job-34 ctx-f9253e30) (logid:baac3e91) Seq 1-1506454075355431213: Sending  { Cmd , MgmtId: 167780532, via: 1(ref-trl-5592-k-Mr8-wei-zhou-kvm1), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":"3","name":"i-2-3-VM"...
...

2023-09-05 16:47:24,815 DEBUG [c.c.a.t.Request] (AgentManager-Handler-20:null) (logid:) Seq 1-1506454075355431213: Processing:  { Ans: , MgmtId: 167780532, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 1a30047b-99dd-4b7b-b2e5-4e862c1117a5 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '1a30047b-99dd-4b7b-b2e5-4e862c1117a5'
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:278)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:264)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDisksViaVmSpec(KVMStoragePoolManager.java:239)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.handleRollback(LibvirtPrepareForMigrationCommandWrapper.java:150)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:62)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:52)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1848)
        at com.cloud.agent.Agent.processRequest(Agent.java:662)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
        at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }

[yadvr]

@weizhouapache I think this may be an env issue or something that affects only local storage post upgrade. If I register a new/fresh template after upgrade and then deploy VM and migrate with (local) storage, it seems to be working. The issue is the MigrationCommand fails and upon failure a prepareformigration command is sent with rollback=true.

Could you test by registering a new template again?

I suspect the I/O issue could be due to something about the migrate command not copying the template (backing file) or something else?

[yadvr]

@weizhouapache update - I downloaded and re-registered the same template in my env, deployed a VM and that can live migrate with storage without any issues 🤦 I suspect if you were to re-register the template you were template - it might just work - now to find out why this is happening.

[yadvr]

Could be related - #5759 cc @weizhouapache

[weizhouapache]

@rohityadavcloud
Thanks for the information

[weizhouapache]

This seems to be related to #7408

@harikrishna-patnala
can you please have a look ?

[yadvr]

Update - for my specific template, it had some io_uring related settings removing those the vm migration seems to be working for newly deployed VMs. It could still be combination of other issues, not a blocker for me now. Thanks @weizhouapache

[weizhouapache]

Update - for my specific template, it had some io_uring related settings removing those the vm migration seems to be working for newly deployed VMs. It could still be combination of other issues, not a blocker for me now. Thanks @weizhouapache

Thanks @rohityadavcloud for you findings.
I will investigate it tomorrow
It would be good if we can at least find a workaround

[weizhouapache]

we have found the root cause

diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/
+resource/LibvirtConnection.java
index c70a72f399c..0f8031e3aaa 100644
--- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java
+++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java
@@ -21,6 +21,7 @@ import java.util.Map;

 import org.apache.log4j.Logger;
 import org.libvirt.Connect;
+import org.libvirt.Library;
 import org.libvirt.LibvirtException;

 import com.cloud.hypervisor.Hypervisor;
@@ -44,6 +45,7 @@ public class LibvirtConnection {
         if (conn == null) {
             s_logger.info("No existing libvirtd connection found. Opening a new one");
             conn = new Connect(hypervisorURI, false);
+            Library.initEventLoop();
             s_logger.debug("Successfully connected to libvirt at: " + hypervisorURI);
             s_connections.put(hypervisorURI, conn);
         } else {

@harikrishna-patnala is working on the fix.