problem when launch pkvm

[n1ptune]

Hi @maluka-dmytro ,
I'm trying to launch pkvm in a qemu vm with this command

sudo ./qemu/build/qemu-system-x86_64 -drive file=debian-13-nocloud-amd64.qcow2,format=qcow2 \
	-machine q35,accel=kvm,kernel-irqchip=split \
	-cpu host,kvm-pv-unhalt=off,kvm-pv-ipi=off,kvm-pv-sched-yield=off \
	-device intel-iommu,aw-bits=48 \
	-m 4G -smp 8 \
	-vga virtio \
	-device e1000,netdev=net0 \
	-netdev user,id=net0,hostfwd=tcp::2222-:22 \
	-serial mon:stdio -s \
	-monitor unix:qemu-monitor-socket,server,nowait \
	-kernel pKVM-IA-host/arch/x86_64/boot/bzImage \
	-append "root=/dev/sda1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200n8 kvm-intel.pkvm=1 nokaslr" \
    -nographic 

pKVM-IA-host/arch/x86_64/boot/bzImage is the host kernel built with CONFIG_PKVM_INTEL=y using branch pvVMCS-POC-v6.12
but it keeps reporting errors like this

[    0.822607] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
[    0.822607] DMAR: QI PRIOR: Invalidation Wait qw0 = 0x200000025, qw1 = 0x100cccc04
[    0.822607] DMAR: VT-d detected Invalidation Queue Error: Reason 0
[    0.822607] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
[    0.822607] DMAR: QI PRIOR: Invalidation Wait qw0 = 0x200000025, qw1 = 0x100cccc04
[    0.822607] DMAR: VT-d detected Invalidation Queue Error: Reason 0
[    0.822607] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
[    0.822607] DMAR: QI PRIOR: Invalidation Wait qw0 = 0x200000025, qw1 = 0x100cccc04
[    0.822607] DMAR: VT-d detected Invalidation Queue Error: Reason 0
[    0.822607] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
[    0.822607] DMAR: QI PRIOR: Invalidation Wait qw0 = 0x200000025, qw1 = 0x100cccc04
[    0.822607] DMAR: VT-d detected Invalidation Queue Error: Reason 0
[    0.822607] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
[    0.822607] DMAR: QI PRIOR: Invalidation Wait qw0 = 0x200000025, qw1 = 0x100cccc04
[    0.822607] DMAR: VT-d detected Invalidation Queue Error: Reason 0
[    0.822607] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0

it seems like that the iommu problem.
So i try not to use kvm but use qemu tcg instead.
But i find qemu tcg don't support vmx

qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID[eax=01h].ECX.vmx [bit 5]

So how do you launch a pkvm host.Do I have to run it on real hardware only?

[maluka-dmytro]

TCG is definitely not the way to go. :)

I run pKVM in qemu with the same parameters, including -device intel-iommu,aw-bits=48, but I haven't seen such failures. I guess it depends on which virtual devices your VM is using.

You might try just removing -device intel-iommu,aw-bits=48?

[n1ptune]

TCG is definitely not the way to go. :)

I run pKVM in qemu with the same parameters, including -device intel-iommu,aw-bits=48, but I haven't seen such failures. I guess it depends on which virtual devices your VM is using.

You might try just removing -device intel-iommu,aw-bits=48?

removing -device intel-iommu,aw-bits=48 will not report error above,but the pkvm also will not launched,because pkvm need iommu support in pKVM-IA-host/arch/x86/kvm/vmx/pkvm/pkvm_host.c function vmx_pkvm_init and pkvm_iommu_driver_prepare,
the pkvm_iommu_driver_prepare will return -ENOENT so vmx_pkvm_init will fail.

[n1ptune]

besides i'm not sure why you are using accel=kvm, besause i think when you use kvm you are launch a L1 guest, but pkvm should work in L0, it is weird

[maluka-dmytro]

removing -device intel-iommu,aw-bits=48 will not report error above,but the pkvm also will not launched,because pkvm need iommu support in pKVM-IA-host/arch/x86/kvm/vmx/pkvm/pkvm_host.c function vmx_pkvm_init and pkvm_iommu_driver_prepare

Yeah true, pKVM requires IOMMU to protect protected VMs memory and its own memory from DMA access by the host.

So you might try commenting out pkvm_iommu_driver_prepare() and pkvm_iommu_driver_init() (just for testing purpose). Honestly I haven't tried that myself, but I guess it should work (just the pKVM's list of IOMMUs would be empty, so pKVM's IOMMU-related operation operations would be no-ops).

besides i'm not sure why you are using accel=kvm, besause i think when you use kvm you are launch a L1 guest, but pkvm should work in L0, it is weird

I'm not sure what you mean. Launching pKVM in qemu is just for development purposes, to simulate a real system. On a real system, the pKVM hypervisor is in L0, the host (with KVM) is in L1, and guest VMs run by the host via crosvm are L2 VMs. When running in qemu instead, the native host with KVM + qemu is L0, the pKVM hypervisor is L1, the "host" is L2, and guest VMs run by the "host" via crosvm are L3 VMs.

Qemu, from what I know, doesn't implement VMX emulation, which is why we cannot use TCG to run pKVM. And even if we could, emulating L0 + L1 + L2 in qemu would be damn slow, right?

[n1ptune]

Yeah true, pKVM requires IOMMU to protect protected VMs memory and its own memory from DMA access by the host.

So you might try commenting out pkvm_iommu_driver_prepare() and pkvm_iommu_driver_init() (just for testing purpose). Honestly I haven't tried that myself, but I guess it should work (just the pKVM's list of IOMMUs would be empty, so pKVM's IOMMU-related operation operations would be no-ops).

it works.

I'm not sure what you mean. Launching pKVM in qemu is just for development purposes, to simulate a real system. On a real system, the pKVM hypervisor is in L0, the host (with KVM) is in L1, and guest VMs run by the host via crosvm are L2 VMs. When running in qemu instead, the native host with KVM + qemu is L0, the pKVM hypervisor is L1, the "host" is L2, and guest VMs run by the "host" via crosvm are L3 VMs.

Qemu, from what I know, doesn't implement VMX emulation, which is why we cannot use TCG to run pKVM. And even if we could, emulating L0 + L1 + L2 in qemu would be damn slow, right?

sorry for confusing you, I misunderstood the concept of VMX virtualization. You are right.

[n1ptune]

disable iommu could launch host but failed to launch guest. Cause device such as APIC relies on iommu,therefore, the guest will remain stuck during boot.

and here is the clear log of use iommu

[    1.082021] pkvm: submit_qi free_head 0x2

[    1.082648] pkvm: submit_qi free_head 0x4

[    1.084208] DMAR: IOMMU batching disallowed due to virtualization

[    1.085208] pci 0000:00:00.0: Adding to iommu group 0

[    1.086002] pci 0000:00:01.0: Adding to iommu group 1

[    1.086726] pci 0000:00:02.0: Adding to iommu group 2

[    1.087431] pci 0000:00:1f.0: Adding to iommu group 3

[    1.088209] pci 0000:00:1f.2: Adding to iommu group 3

[    1.088959] pci 0000:00:1f.3: Adding to iommu group 3

qemu-system-x86_64: vtd_process_inv_desc: invalid inv desc: hi=f000ff53f000e2c3, lo=f000ff53f000ff53 (unknow)

[    1.089914] DMAR: VT-d detected Invalidation Queue Error: Reason 0

[    1.089915] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0

[    1.090179] DMAR: DRHD: handling fault status reg 10

[    1.090697] DMAR: QI PRIOR: Invalidation Wait qw0 = 0x200000025, qw1 = 0x10156e804

[    1.090697] DMAR: VT-d detected Invalidation Queue Error: Reason 0

[    1.090697] DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0

[    1.090697] DMAR: QI PRIOR: Invalidation Wait qw0 = 0x200000025, qw1 = 0x10156e804

so i guess it's the communication between iommu and pkvm get someting wrong.
Did you use a specific qemu version to launch the host or just default?

[maluka-dmytro]

No, I'm not just using a specific qemu version, I'm using the default one. (BTW for launching guest VMs with pKVM enabled, if we want to launch them via qemu instead of crosvm, currently we do need a custom qemu build for that. But that's another story.)

I've just quickly tried this change on top of pvVMCS-POC-v6.12 branch:

--- a/arch/x86/kvm/vmx/pkvm/pkvm_host.c
+++ b/arch/x86/kvm/vmx/pkvm/pkvm_host.c
@@ -1772,6 +1772,7 @@ int __init vmx_pkvm_init(void)
 		goto fw_rmem_clear;
 	}
 
+#if 0
 	/*
 	 * IOMMU initialization is spread across multiple places
 	 * in the host. pkvm expects the drhd structures to be filled
@@ -1781,6 +1782,7 @@ int __init vmx_pkvm_init(void)
 	ret = pkvm_iommu_driver_prepare();
 	if (ret)
 		goto fw_rmem_clear;
+#endif
 
 	ret = __vmx_pkvm_init();
 
@@ -1789,11 +1791,15 @@ int __init vmx_pkvm_init(void)
 	else
 		pkvm_init_rollback();
 
+#if 0
 	/*
 	 * Initialize iommu regardless of whether pkvm initialization
 	 * succeeded or not.
 	 */
 	iommu_ret = pkvm_iommu_driver_init();
+#else
+	iommu_ret = 0;
+#endif
 
 	if (!ret) {
 		if (iommu_ret) {

and it is successfully booting for me with qemu without iommu. It even somehow successfully enables x2APIC mode (even though AFAIK x2APIC does depend on IRQ remapping and thus on IOMMU...)

[n1ptune]

i mean it is ok to launch host with pkvm when commenting this two function but failed to launch guest using crosvm.Have you ever tried to launch guest with a host commenting this two function？ 发件人："maluka-dmytro" ***@***.***> 发送日期：2025-11-26 23:03:06 收件人："intel-staging/pKVM-IA" ***@***.***> 抄送人： 主 题：Re: [intel-staging/pKVM-IA] problem when launch pkvm (Issue #47) maluka-dmytro left a comment (intel-staging/pKVM-IA#47) No, I'm not just using a specific qemu version, I'm using the default one. (BTW for launching guest VMs with pKVM enabled, if we want to launch them via qemu instead of crosvm, currently we do need a custom qemu build for that. But that's another story.) I've just quickly tried this change on top of pvVMCS-POC-v6.12 branch:

…

--- a/arch/x86/kvm/vmx/pkvm/pkvm_host.c +++ b/arch/x86/kvm/vmx/pkvm/pkvm_host.c @@ -1772,6 +1772,7 @@ int __init vmx_pkvm_init(void) goto fw_rmem_clear; } +#if 0 /* * IOMMU initialization is spread across multiple places * in the host. pkvm expects the drhd structures to be filled @@ -1781,6 +1782,7 @@ int __init vmx_pkvm_init(void) ret = pkvm_iommu_driver_prepare(); if (ret) goto fw_rmem_clear; +#endif ret = __vmx_pkvm_init(); @@ -1789,11 +1791,15 @@ int __init vmx_pkvm_init(void) else pkvm_init_rollback(); +#if 0 /* * Initialize iommu regardless of whether pkvm initialization * succeeded or not. */ iommu_ret = pkvm_iommu_driver_init(); +#else + iommu_ret = 0; +#endif if (!ret) { if (iommu_ret) { and it is successfully booting for me with qemu without iommu. It even somehow successfully enables x2APIC mode (even though AFAIK x2APIC does depend on IRQ remapping and thus on IOMMU...) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[maluka-dmytro]

Yep, I tried running e.g. an Ubuntu Cloud VM with the following command:

crosvm run --disable-sandbox \
	--mem 2048 --cpus $(nproc) \
	--serial type=stdout,stdin=true,console=true,earlycon=true \
	--block noble-server-cloudimg-amd64.raw \
	--params "root=/dev/vda1" \
	bzImage

and it boots successfully. And if I add --protected-vm-without-firmware to run it protected, it also boots successfully.

[n1ptune]

Still don't work for me. I moved the native host physical machine from an Intel(R) Xeon(R) Platinum 8468 CPU to an Intel(R) Xeon(R) Gold 6132 CPU, but the same error still occurs.
Is there any other kernel config I should pick in host besides PKVM_INTEL and in guest besides PKVM_GUEST?

[maluka-dmytro]

Still don't work for me. I moved the native host physical machine from an Intel(R) Xeon(R) Platinum 8468 CPU to an Intel(R) Xeon(R) Gold 6132 CPU, but the same error still occurs. Is there any other kernel config I should pick in host besides PKVM_INTEL and in guest besides PKVM_GUEST?

Not that I remember of.

I'm using, for example, the attached config-debian-trixie-6_12-pkvm.txt for host kernel, and
config-ubuntu-cloud-noble-6_12-pkvmguest.txt for guest kernel.