Error deleting VM. ACS 4.22.0.0 KVM Ceph #12226

sptest232 · 2025-12-10T14:11:58Z

sptest232
Dec 10, 2025

ACS 4.22.0.0, KVM host OS - Oracle Linux Server 10.1 (Agent version - 4.22.0.0), ceph (20.2.0 tentacle).
An error occurs when deleting a VM.

2025-12-10 16:34:15,666 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) VM instance {"id":6959,"instanceName":"i-332-6959-VM","state":"Stopping","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"} state transited from [Stopping] to [Stopping] with event [StopRequested]. VM's original host: Host {"id":77387,"name":"kvm-sr246","type":"Routing","uuid":"8843bce1-e11e-4f96-9dcf-6355f723922f"}, new host: Host {"id":77387,"name":"kvm-sr246","type":"Routing","uuid":"8843bce1-e11e-4f96-9dcf-6355f723922f"}, host before state transition: Host {"id":77387,"name":"kvm-sr246","type":"Routing","uuid":"8843bce1-e11e-4f96-9dcf-6355f723922f"}
2025-12-10 16:34:15,713 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[]) (logid:) Seq 77387-94012642221359648: Processing: { Ans: , MgmtId: 108597816265214, via: 77387, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.IllegalArgumentException: No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskCacheMode.
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at com.cloud.hypervisor.kvm.resource.LibvirtVMDef$DiskDef$DiskCacheMode.valueOf(LibvirtVMDef.java:709)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:114)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getDisks(LibvirtComputingResource.java:4818)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getVmDiskStat(LibvirtComputingResource.java:4883)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmDiskStatsCommandWrapper.execute(LibvirtGetVmDiskStatsCommandWrapper.java:50)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmDiskStatsCommandWrapper.execute(LibvirtGetVmDiskStatsCommandWrapper.java:36)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-10 16:34:15,713 DEBUG [c.c.a.t.Request] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Seq 77387-94012642221359648: Received: { Ans: , MgmtId: 108597816265214, via: 77387(kvm-sr246), Ver: v1, Flags: 10, { Answer } }
2025-12-10 16:34:15,714 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Details from executing class com.cloud.agent.api.GetVmDiskStatsCommand: java.lang.IllegalArgumentException: No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskCacheMode.
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at com.cloud.hypervisor.kvm.resource.LibvirtVMDef$DiskDef$DiskCacheMode.valueOf(LibvirtVMDef.java:709)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:114)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getDisks(LibvirtComputingResource.java:4818)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getVmDiskStat(LibvirtComputingResource.java:4883)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmDiskStatsCommandWrapper.execute(LibvirtGetVmDiskStatsCommandWrapper.java:50)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmDiskStatsCommandWrapper.execute(LibvirtGetVmDiskStatsCommandWrapper.java:36)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-10 16:34:15,714 WARN [c.c.v.UserVmManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Error while collecting disk stats for vm: VM instance {"id":695
9,"instanceName":"i-332-6959-VM","state":"Stopping","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"} from host: Host {"id":77387,"name":"kvm-sr246","type":"Routing","uuid":"8843bce1-
e11e-4f96-9dcf-6355f723922f"} java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.GetVmDiskStatsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.Ge
tVmDiskStatsAnswer are in unnamed module of loader 'app')
at com.cloud.vm.UserVmManagerImpl.collectVmDiskStatistics(UserVmManagerImpl.java:6023)
at com.cloud.vm.UserVmManagerImpl.collectVmDiskAndNetworkStatistics(UserVmManagerImpl.java:9924)
at com.cloud.vm.UserVmManagerImpl.collectVmDiskAndNetworkStatistics(UserVmManagerImpl.java:9916)
at com.cloud.vm.UserVmManagerImpl.prepareStop(UserVmManagerImpl.java:9293)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:2502)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:2355)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5970)
at jdk.internal.reflect.GeneratedMethodAccessor1474.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-12-10 16:34:15,768 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 77387-94012642221359649: Processing: { Ans: , MgmtId: 108597816265214, via: 77387, Ver: v1, Flags: 10, [{"com.cloud.agent
.api.Answer":{"result":"false","details":"java.lang.IllegalArgumentException: No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskCacheMode.
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at com.cloud.hypervisor.kvm.resource.LibvirtVMDef$DiskDef$DiskCacheMode.valueOf(LibvirtVMDef.java:709)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:114)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getInterfaces(LibvirtComputingResource.java:4796)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getVmNetworkStat(LibvirtComputingResource.java:4859)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmNetworkStatsCommandWrapper.execute(LibvirtGetVmNetworkStatsCommandWrapper.java:50)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmNetworkStatsCommandWrapper.execute(LibvirtGetVmNetworkStatsCommandWrapper.java:36)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-10 16:34:15,768 DEBUG [c.c.a.t.Request] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Seq 77387-94012642221359649: Received: { Ans: , MgmtId: 10859781626521
4, via: 77387(kvm-sr246), Ver: v1, Flags: 10, { Answer } }
2025-12-10 16:34:15,768 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Details from executing class com.cloud.agent.api.GetV
mNetworkStatsCommand: java.lang.IllegalArgumentException: No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskCacheMode.

at java.base/java.lang.Enum.valueOf(Enum.java:293)
at com.cloud.hypervisor.kvm.resource.LibvirtVMDef$DiskDef$DiskCacheMode.valueOf(LibvirtVMDef.java:709)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:114)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getInterfaces(LibvirtComputingResource.java:4796)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getVmNetworkStat(LibvirtComputingResource.java:4859)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmNetworkStatsCommandWrapper.execute(LibvirtGetVmNetworkStatsCommandWrapper.java:50)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVmNetworkStatsCommandWrapper.execute(LibvirtGetVmNetworkStatsCommandWrapper.java:36)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-10 16:34:15,768 WARN [c.c.v.UserVmManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Error while collecting network stats for vm: VM instance {"id":
6959,"instanceName":"i-332-6959-VM","state":"Stopping","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"} from host: Host {"id":77387,"name":"kvm-sr246","type":"Routing","uuid":"8843bc
e1-e11e-4f96-9dcf-6355f723922f"} java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.GetVmNetworkStatsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.
api.GetVmNetworkStatsAnswer are in unnamed module of loader 'app')
at com.cloud.vm.UserVmManagerImpl.collectVmNetworkStatistics(UserVmManagerImpl.java:5142)
at com.cloud.vm.UserVmManagerImpl.collectVmDiskAndNetworkStatistics(UserVmManagerImpl.java:9925)
at com.cloud.vm.UserVmManagerImpl.collectVmDiskAndNetworkStatistics(UserVmManagerImpl.java:9916)
at com.cloud.vm.UserVmManagerImpl.prepareStop(UserVmManagerImpl.java:9293)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:2502)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:2355)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5970)
at jdk.internal.reflect.GeneratedMethodAccessor1474.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-12-10 16:34:15,780 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) VM instance {"id":6959,"instanceName":"i-332-6
959-VM","state":"Stopping","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"} is a [User], returning null for control Nic IP.
2025-12-10 16:34:15,780 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Wait time setting on com.cloud.agent.api.StopCommand
is 1800 seconds
2025-12-10 16:34:15,780 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Seq 77394-94012642221359650: Routed from 108597816265214
2025-12-10 16:34:15,781 DEBUG [c.c.a.t.Request] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Seq 77387-94012642221359650: Sending { Cmd , MgmtId: 108597816265214,
via: 77387(kvm-sr246), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","forceStop":"false","vlanToPersistenceMap":{"2250":"true"},"exp
ungeVM":"false","volumesToDisconnect":[],"vmName":"i-332-6959-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-10 16:34:15,803 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-756eb558]) (logid:1943dd78) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds
2025-12-10 16:34:15,804 DEBUG [c.c.a.m.ClusteredDirectAgentAttache] (StatsCollector-6:[ctx-756eb558]) (logid:1943dd78) Seq 77394-6782983988773400411: Routed from 108597816265214
2025-12-10 16:34:15,804 DEBUG [c.c.a.m.D.Task] (DirectAgent-106:[ctx-7d46979e]) (logid:838c6948) Seq 77394-6782983988773400411: Executing request
2025-12-10 16:34:15,824 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 77387-94012642221359650: Processing: { Ans: , MgmtId: 108597816265214, via: 77387, Ver: v1, Flags: 10, [{"com.cloud.agent
.api.Answer":{"result":"false","details":"java.lang.IllegalArgumentException: No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskCacheMode.
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at com.cloud.hypervisor.kvm.resource.LibvirtVMDef$DiskDef$DiskCacheMode.valueOf(LibvirtVMDef.java:709)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:114)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getDisks(LibvirtComputingResource.java:4818)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:87)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:47)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
","wait":"0","bypassHostMaintenance":"false"}}] }
2025-12-10 16:34:15,824 DEBUG [c.c.a.t.Request] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Seq 77387-94012642221359650: Received: { Ans: , MgmtId: 10859781626521
4, via: 77387(kvm-sr246), Ver: v1, Flags: 10, { Answer } }
2025-12-10 16:34:15,824 WARN [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Unable to stop vm VM instance {"id":6959,"inst
anceName":"i-332-6959-VM","state":"Stopping","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"}
2025-12-10 16:34:15,831 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) VM instance {"id":6959,"instanceName":"i-332-6959-VM","state"
:"Running","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"} state transited from [Stopping] to [Running] with event [OperationFailed]. VM's original host: Host {"id":77387,"name":"kvm-sr246-u1617.aisr
p.local","type":"Routing","uuid":"8843bce1-e11e-4f96-9dcf-6355f723922f"}, new host: Host {"id":77387,"name":"kvm-sr246","type":"Routing","uuid":"8843bce1-e11e-4f96-9dcf-6355f723922f"}, host befo
re state transition: Host {"id":77387,"name":"kvm-sr246","type":"Routing","uuid":"8843bce1-e11e-4f96-9dcf-6355f723922f"}
2025-12-10 16:34:15,832 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Invocation exception, caused by: com.cloud.utils.exception.
CloudRuntimeException: Unable to stop VM instance {"id":6959,"instanceName":"i-332-6959-VM","state":"Running","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"}
2025-12-10 16:34:15,832 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719, ctx-5fcf1916]) (logid:56c78da9) Rethrow exception com.cloud.utils.exception.CloudRuntimeExc
eption: Unable to stop VM instance {"id":6959,"instanceName":"i-332-6959-VM","state":"Running","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"}
2025-12-10 16:34:15,832 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 6959, job orig
in: 124718
2025-12-10 16:34:15,832 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Unable to complete AsyncJob {"accountId":36,"cmd":"com.cloud.vm.VmWorkStop"
,"cmdInfo":"rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4
cAAAAAAAAAAkAAAAAAAAACcAAAAAAAAbL3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA","cmdVersion":0,"completeMsid":null,"created":"Wed Dec 10 16:34:15 MSK 2025","id":124719,"initMsid":108597816265214,"instanceId":null,"inst
anceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":39,"uuid":"816c3866-50ab-4d22-bcb2-6ebb379421ed"}, job origin: 124
718 com.cloud.utils.exception.CloudRuntimeException: Unable to stop VM instance {"id":6959,"instanceName":"i-332-6959-VM","state":"Running","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"}
at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:2551)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:2355)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5970)
at jdk.internal.reflect.GeneratedMethodAccessor1474.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-12-10 16:34:15,836 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Publish async job-124719 complete on message bus
2025-12-10 16:34:15,836 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Wake up jobs related to job-124719
2025-12-10 16:34:15,836 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Update db status for job-124719
2025-12-10 16:34:15,837 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Wake up jobs joined with job-124719 and disjoin all subjobs created f
rom job- 124719
2025-12-10 16:34:15,845 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Done executing com.cloud.vm.VmWorkStop for job-124719
2025-12-10 16:34:15,846 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-144:[ctx-c925c4b8, job-124718/job-124719]) (logid:56c78da9) Remove job-124719 from job monitoring
2025-12-10 16:34:15,862 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-40:[ctx-ef1768c5, job-124718]) (logid:56c78da9) Unexpected exception while executing org.apache.cloudstack.api.command.admin.account.D
isableAccountCmd com.cloud.utils.exception.CloudRuntimeException: Unable to stop VM instance {"id":6959,"instanceName":"i-332-6959-VM","state":"Running","type":"User","uuid":"70847e83-b8d3-4116-bb2c-1e32079a8f0f"
}
at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:2551)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:2355)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5970)
at jdk.internal.reflect.GeneratedMethodAccessor1474.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-12-10 16:34:15,863 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-40:[ctx-ef1768c5, job-124718]) (logid:56c78da9) Complete async job-124718, jobStatus: FAILED, resultCode: 530, result: org.apache.
cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Unable to stop VM instance {"id":6959,"instanceName":"i-332-6959-VM","state":"Running","type":"User","uuid":"70847e83-b
8d3-4116-bb2c-1e32079a8f0f"}"}
2025-12-10 16:34:16,007 ERROR [o.a.c.f.m.MessageBusBase] (pool-17015-thread-1:[]) (logid:) NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION! com.cloud.utils.exception.CloudRuntimeException: NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
at org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageBusBase.java:168)
at com.cloud.capacity.CapacityManagerImpl.updateCapacityForHost(CapacityManagerImpl.java:786)
at com.cloud.alert.AlertManagerImpl$1.doInTransactionWithoutResult(AlertManagerImpl.java:320)
at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
at com.cloud.alert.AlertManagerImpl.lambda$recalculateHostCapacities$0(AlertManagerImpl.java:316)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

==========================================================
agent.log:

WARN [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:fd97e4d1) Caught: java.lang.IllegalArgumentException: No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskCacheMode.
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at com.cloud.hypervisor.kvm.resource.LibvirtVMDef$DiskDef$DiskCacheMode.valueOf(LibvirtVMDef.java:709)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:114)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getVncPort(LibvirtComputingResource.java:4668)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVncPortCommandWrapper.execute(LibvirtGetVncPortCommandWrapper.java:41)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVncPortCommandWrapper.execute(LibvirtGetVncPortCommandWrapper.java:32)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

========================================================

agent.err:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
libvirt: Storage Driver error : Storage pool not found: no storage pool with matching uuid 'c0d74da4-e790-42a6-b44d-e395ba50d1bf'
libvirt: Storage Driver error : Storage pool not found: no storage pool with matching uuid '234a8740-7c52-3a83-b995-e2a7b85f2967'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'i-287-6943-VM'
libvirt: error : no connection driver available for lxc:///
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'i-332-6952-VM'
libvirt: error : no connection driver available for lxc:///
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '6beeb063-6461-4b44-8442-67bc99cc13ca' (i-287-6943-VM)
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'i-287-6943-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '9d280bb8-218b-49be-9afe-2d434467c516' (i-332-6952-VM)
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'i-332-6952-VM'
libvirt: Storage Driver error : Storage volume not found: no storage vol with matching name 'b2b05820-6c0a-4bc3-a3a1-9c792c12c678'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'i-332-6959-VM'
libvirt: Storage Driver error : Storage volume not found: no storage vol with matching name 'b4f08622-7459-4860-8aad-2e50cbafc761'
libvirt: Storage Driver error : Storage volume not found: no storage vol with matching name 'cb822720-8ceb-4601-b104-073f065eb61c'
libvirt: Storage Driver error : failed to remove volume 'sr241-u5-u14-1/03b56a1a-c2d6-41db-ad3b-ecadc7fa448a': No such file or directory
libvirt: Storage Driver error : failed to remove volume 'sr241-u5-u14-1/b4f08622-7459-4860-8aad-2e50cbafc761': No such file or directory
libvirt: Storage Driver error : Storage volume not found: no storage vol with matching name '480f1b9a-82ab-4278-96c3-6ceb02416e05'
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

========================

agent.out
WARNING: JAVA version is 21. JAVA 17 is recommended.
WARNING: JAVA version is 21. JAVA 17 is recommended.
WARNING: JAVA version is 21. JAVA 17 is recommended.

Answered by sptest232

Dec 12, 2025

Did you try this @sptest232 : If the vm is still running on the kvm host , you can use virsh command to shutdown or destroy the instance, and restart cloudstack-agent. After you did this, you can force delete the VM.

The bug you hit prevents doing it from cloudstack. Once the fix at #12248 is in, doing it from ACS should not be an issue anymore.

I deleted the VM another way:

enabled maintenance mode on the RBD PS.
After enabling maintenance mode, the VM status changed from "Stopping" to "Stopped".
deleted the VM from the UI.
the disks were removed by making changes to DB ACS (UPDATE volumes set removed=NOW() WHERE id=xxx;).

View full answer

DaanHoogland · 2025-12-10T14:28:41Z

DaanHoogland
Dec 10, 2025
Collaborator

@sptest232 , the java version of (or otherwise a classpath) is definately part of the problem as public class GetVmDiskStatsAnswer extends Answer is the definition of the class not being cast correctly.

as for the java.lang.IllegalArgumentException: No enum constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskCacheMode, can you give the xml def of the vm you are trying to delete? An element cache=‘<diskCacheMode>’ should be there.

15 replies

weizhouapache Dec 12, 2025
Collaborator

Yes, the volume exists, but I want to delete the RDB Primary Storage (for reinstallation) from CloudStack, I don’t need the information that is currently stored there, but I can’t while the ACS contains information about the volumes stored on this PS.

if you have database access,

UPDATE volumes set removed=NOW() WHERE id=xxx;

DaanHoogland Dec 12, 2025
Collaborator

Did you try this @sptest232 : If the vm is still running on the kvm host , you can use virsh command to shutdown or destroy the instance, and restart cloudstack-agent.
After you did this, you can force delete the VM.

The bug you hit prevents doing it from cloudstack. Once the fix at #12248 is in, doing it from ACS should not be an issue anymore.

sptest232 Dec 12, 2025
Author

Yes, the volume exists, but I want to delete the RDB Primary Storage (for reinstallation) from CloudStack, I don’t need the information that is currently stored there, but I can’t while the ACS contains information about the volumes stored on this PS.

if you have database access,
UPDATE volumes set removed=NOW() WHERE id=xxx;

Thank you very much!

sptest232 Dec 12, 2025
Author

Did you try this @sptest232 : If the vm is still running on the kvm host , you can use virsh command to shutdown or destroy the instance, and restart cloudstack-agent. After you did this, you can force delete the VM.

The bug you hit prevents doing it from cloudstack. Once the fix at #12248 is in, doing it from ACS should not be an issue anymore.

I deleted the VM another way:

enabled maintenance mode on the RBD PS.
After enabling maintenance mode, the VM status changed from "Stopping" to "Stopped".
deleted the VM from the UI.
the disks were removed by making changes to DB ACS (UPDATE volumes set removed=NOW() WHERE id=xxx;).

Answer selected by DaanHoogland

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Error deleting VM. ACS 4.22.0.0 KVM Ceph #12226

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 15 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Error deleting VM. ACS 4.22.0.0 KVM Ceph #12226

Uh oh!

sptest232 Dec 10, 2025

Replies: 1 comment · 15 replies

Uh oh!

DaanHoogland Dec 10, 2025 Collaborator

Uh oh!

weizhouapache Dec 12, 2025 Collaborator

Uh oh!

DaanHoogland Dec 12, 2025 Collaborator

Uh oh!

sptest232 Dec 12, 2025 Author

Uh oh!

sptest232 Dec 12, 2025 Author

sptest232
Dec 10, 2025

Replies: 1 comment 15 replies

DaanHoogland
Dec 10, 2025
Collaborator

weizhouapache Dec 12, 2025
Collaborator

DaanHoogland Dec 12, 2025
Collaborator

sptest232 Dec 12, 2025
Author

sptest232 Dec 12, 2025
Author