diff --git a/docs/ioregionfd.md b/docs/ioregionfd.md index a0cb7d73..8416435c 100644 --- a/docs/ioregionfd.md +++ b/docs/ioregionfd.md @@ -4,16 +4,19 @@ ioregionfd is a mechanism that speeds up ioeventfds: https://lore.kernel.org/kvm/cover.1613828726.git.eafanasova@gmail.com/. In the author's original words: "ioregionfd is a KVM dispatch mechanism which can be used for handling MMIO/PIO accesses over file descriptors without returning -from ioctl(KVM_RUN).". +from `ioctl(KVM_RUN)`.". It hasn't yet been merged upstream and it's not +supported by libvfio-user. -libvfio-user currently supports an experimental variant of this mechanism -called shadow ioeventfd. A shadow ioeventfd is a normal ioeventfd where the -vfio-user server passes another piece of memory (called the _shadow_ memory) -via an additional file descriptor when configuring the ioregionfd, which then -QEMU memory maps and passes this address to KVM. This shadow memory is never -exposed to the guest. When the guest writes to the trapped memory, KVM writes -the value to the shadow memory instread of discarding it, and then proceeds -kicking the eventfd as normal. +# shadow ioeventfd + +libvfio-user currently supports a similar mechanism called shadow ioeventfd. +A shadow ioeventfd is a normal ioeventfd where the vfio-user server passes +another piece of memory (called the _shadow_ memory) via an additional file +descriptor when configuring the ioregionfd, which then QEMU memory maps and +passes this address to KVM. This shadow memory is never exposed to the guest. +When the guest writes to the trapped memory, KVM writes the value to the shadow +memory instead of discarding it, and then proceeds kicking the eventfd as +normal. To use shadow ioeventfd, the kernel and QEMU need to be patched. The QEMU patch is designed specifically for SPDK's doorbells (one ioregionfd of 4K in BAR0); @@ -21,15 +24,15 @@ it should be trivial to extend. The list of patches: * kernel: https://gist.github.com/tmakatos/532afd092a8df2175120d3dbfcd719ef -* QEMU: https://gist.github.com/tmakatos/57755d2a37a6d53c9ff392e7c34470f6 +* QEMU: https://github.com/tmakatos/qemu/tree/shadow-ioeventfd * SPDK: https://gist.github.com/tmakatos/f6c10fdaff59c9d629f94bd8e44a53bc shadow ioeventfd sample ----------------------- -samples/shadow_ioeventfd_server.c implements a vfio-user server that allows a +`samples/shadow_ioeventfd_server.c` implements a vfio-user server that allows a part of its BAR0 to be accessed via a shadow ioeventfd. -shadow_ioeventfd_speed_test.c is run in the guest. It compares peformance of +`shadow_ioeventfd_speed_test.c` is run in the guest. It compares performance of shadow ioeventfd vs. vfio-user messages by repeatedly writing to the part of the BAR0 that is handled by shadow ioeventfd and to the part not handled by shadow ioeventfd. @@ -38,7 +41,7 @@ To run the sample: * Patch and build the kernel and QEMU using above patches. * Enable support for shadow ioeventfd in libvfio-user (set `shadow-ieoventfd` to `true` in meson_options.txt and then build libvfio-user. -* Run samples/shadow_ioeventfd_server, e.g. +* Run `samples/shadow_ioeventfd_server`, e.g. ``` build/samples/shadow_ioeventfd_server /var/run/cntrl ``` diff --git a/docs/shadow_ioeventfd.md b/docs/shadow_ioeventfd.md new file mode 100644 index 00000000..1f575464 --- /dev/null +++ b/docs/shadow_ioeventfd.md @@ -0,0 +1,15 @@ +# Shadow ioeventfd + +Shadow ioeventfd is mechanism that reduces the cost of MMIO writes in +vfio-user. In a nutshell, it eliminates the involvement of the VMM by allowing +KVM to write the MMIO value in a piece of memory provided by the device +emulation task and then notifying it by kicking the ioeventfd. The device +emulation task can then find the value in a known location. + +This mechanism is especially important for cases where the MMIO value is +required by the device protocol, which is the case for NVMe. + +This functionality requires patching the kernel (KVM) and QEMU: +- kernel patches: https://github.com/tmakatos/linux/tree/shadow-ioeventfd +- QEMU patches: https://github.com/tmakatos/qemu/tree/shadow-ioeventfd +