I have been facing a hard to fix issue in the last few days: very low performance when running some applications inside a virtual machine when using AMD processors. I have two machines connected to each other using an AMD Epyc 7452 32-core processor. In each of these machines I assign 20 cores to a VM using QEMU and then start a server that uses 12 threads in one machine and and a client that uses 12 threads on the other machine. The client sends RPC requests with 1024 byte messages to the server and I measure the throughput per second. I have been seeing a huge decrease in performance when running things inside a VM. For example, I get 36 Gbps when running the client and server on the bare-machine and only 8 Gbps when running inside a VM. In comparison, when running the same configuration using an Intel CPU I get 32 Gbps on the bare machine, actually worse than with the AMD Epyc CPU, but a much improved 24 Gbps when running the client and server inside the VM.
The overhead with the AMD CPU is much higher than anticipated when running things inside a VM, especially since I am externalizing the network stack from the guest VM and running it on the host: the guest VM deposits packets in a shared memory region (QEMU IVSHM) using PCI-passthrough, so I was expecting very minimal I/O overhead. There seemed to be an issue with non-write combining mappings (https://community.amd.com/t5/server-gurus-discussions/amd-terrible-performance-with-qemu-ivshmem/m-p...), so I tried mapping the bar for the shared memory region through resourceX_wc. This improved performance a bit (8 Gbps -> 10 Gbps) but it's still far off the expected ~24Gbps, so I think something is still not configured properly. Has anyone faced similar virtualization issues with AMD processors? Does anyone know of any potential fixes or misconfigurations?
This is the command I used to start my VM with QEMU:
qemu-system-x86_64 \
-nographic -monitor none -serial stdio \
-machine accel=kvm,type=q35 \
-cpu host \
-smp 20 \
-m 10G \
-device virtio-net-pci,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::2220-:22 \
-chardev socket,path="/run/agent",id="agent" \
-device ivshmem-doorbell,vectors=1,chardev="agent" \
-drive if=virtio,format=raw,file=base.img \
-drive if=virtio,format=raw,file=seed.img
KVM seems to be enabled
lsmod | grep kvm
kvm_amd 155648 0
kvm 1032192 1 kvm_amd
ccp 106496 1 kvm_amd
