Using a PCI graphics card in KVM/QEMU on Debian Stretch

Published in in Tech. Last updated: Sun 19 April 2020
#debian #kvm #pci passthrough #qemu #virtualization

Update 2020: follow-up article in regard to PulseAudio can be found here: „Fixing PulseAudio sound problems in QEMU/KVM VMs by building QEMU 4.2 on Debian Buster“.

Why? Some software needs to use a dedicated graphics card, like GPU computation or playing video games. But not everything is available in Debian, so we can use virtual machines to use other operating systems as well, without having to use dual boot.

My setup is as follows: Intel Core i5 processor with integrated graphics, VT-d and IOMMU, and a NVIDIA GeForce GTX dedicated graphics card. These two are necessary for what we are going to do. In a normal scenario you would use your graphics card as the main display output device. You plug it into your PCIe slot, set the card as default in your BIOS and plug in your monitors into the HDMI/DVI outputs of your card. Everything that is done graphically - playing, watching movies, GPU computing - is then done by the card.

In the new setup I want my card to be passed into a virtual machine exclusively. That means my host system (the operating system which boots when I power on my computer, Debian Stretch) cannot make use of it and must use the internal graphics processor (IGP). This setting can be found in your BIOS/UEFI settings menu at start time. You can use two cables from both outputs into one monitor and switch by pressing your source button. Passing in the PCI device has one big advantage over using the emulated graphics device inside a virtual machine: there is no abstraction layer in between, resulting in the best performance.

I did this whole procedure some years ago already (with Wheezy…), but the documentation and software support for passthrough has improved enormously. You should only continue if you know what you are doing. Backup your data and probably have another computer in standby. If you are unsure about the procedure, stop now and ask someone who you think can help you. I'm in no way responsible for any damages to your hardware or software.

Let's begin with the configuration. For the details I advise you to read the extensive article by Heiko Sieger.

  1. Check if your hardware supports IOMMU and VT-d
  2. Check that you are using the correct output (the integrated graphics processor)
  3. Blacklist modprobe modules which could claim the dedicated PCI card (nouveau, nvidia) and add the vfio-pci module with options (ids=)
  4. Add the vfio modules to your initramfs and use update-initramfs -u to create the new initramfs
  5. Reboot

You should now have bound the card to the vfio-pci driver. You can use dmesg | grep vfio to check for existing vfio_pci: add strings which should be the IDs of your video card.

Update 2018-06-24: if you are switching from a NVIDIA PCI graphics card to Intel i915 integrated graphics, your Xorg might need a config in /etc/X11/xorg.conf. My X11 on Debian segfaults after removing all NVIDIA driver related packages (and blacklisting the nouveau kernel module) because it still wanted to use card0 and card1, but card1 is the vfio-bound PCI card (you can check this by comparing the PCI slot identifiers). Login as root and run X -configure. A new autogenerated xorg.conf.new appears in your directory. Remove all sections that might cause trouble (Section "Device" with Identifier "Card1") and restart your X server.

libvirt

In the following I am using libvirt to manage my virtual machines. Disable all display output in your configuration, so that no Spice or VNC are left.

For networking I decided to stay with the default NAT adapter. I know, I know, very bad. In an advanced setup you can create a bridge and connect it to your VM, but I have not experienced any network limits inside the VM yet, so I did not test another configuration.

What you want to do however is using the virtio driver whereever possible. This should be doable with networking and storage. If your guest will be a Windows machine, you need to load the virtio drivers as a removable disk when setting things up. Else your virtual hardware will not show up.

You should use the OVMF BIOS option when creating the machine, because the support for this setup is dependant on UEFI. It helps to create a writable directory where the OVMF_VARS.fd file can be stored, so your VM UEFI settings are persistently saved (e.g. the boot order).

So far so good. For reference, here are the most important snippets from my VM config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
…
<os>
  <type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
  <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
  <!-- use a writable file to save UEFI vars (boot order) -->
  <nvram>/home/bitkeks/.libvirt/win8_vars.fd</nvram>
</os>
<features>
  …
  <kvm>
    <!-- hide virtualisation -->
    <hidden state='on'/>
  </kvm>
</features>
…
<devices>
  <disk type='block' device='disk'>
    <!-- use a LVM logical volume as the hard disk -->
    <driver name='qemu' type='raw' cache='none' io='native'/>
    <source dev='/dev/mapper/vg1_lv-vm'/>
    <target dev='vda' bus='virtio'/>
    <boot order='1'/>
  </disk>
</devices>
…
<qemu:commandline>
  <!-- additional flag to hide KVM -->
  <qemu:arg value='-cpu'/>
  <qemu:arg value='host,kvm=off'/>

  <!-- additional environment variables to connect to PulseAudio session -->
  <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
  <qemu:env name='QEMU_PA_SERVER' value='/tmp/pa.socket'/>
</qemu:commandline>

If you have problems with the GTX drivers in a Windows guest, make sure you have both KVM hiding switches enabled: <kvm><hidden state='on'/></kvm> and -cpu host,kvm=off. Otherwise it could happen that the Windows driver shuts down the hardware device because the device detected the virtual environment and raises errors.

Using PulseAudio for sound

As you can see in my configuration snippet above, I am using an existing PulseAudio session I attach the VM sound device to. This needs some additional configuration: starting the machine via libvirt uses the default system user libvirt-qemu. This can be tricky in combination with the running PulseAudio session which is most likely started by your local user, with permissions only granted to this account.

After my initial setup, I tried using the existing session at /run/user/1000/pulse/native, but the following error appeared in /var/log/libvirt/<vmname>.log during boot:

1
2
3
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init 'pa' audio driver

This error means libvirt-qemu tried to use the referenced session socket but was refused because of permissions. I found two options to solve this problems: either you run QEMU as your session user or you configure PulseAudio to create a UNIX socket with anonymous authentication. Some forum posts suggest to copy the PulseAudio cookie file, but this did not work for me.

For the first option, start your VM as another user by editing /etc/libvirt/qemu.conf:

1
2
3
4
nographics_allow_host_audio = 1
…
user = "bitkeks"
group = "bitkeks"

The second option is probably the better solution, because it does not mix the libvirt-qemu user with your local user account. To create a second socket, copy /etc/pulse/default.pa to ~/.pulse/default.pa (or edit the file in your home directory if it already exists) and add the following line at the bottom.

1
load-module module-native-protocol-unix auth-anonymous=1 socket=/tmp/pa.socket

Restart your PulseAudio daemon with pulseaudio -k and pulseaudio --start. Your volume controls should work as before and PulseAudio should have created both sockets, the one in /run/user/1000/pulse/native and /tmp/pa.socket. The next time you boot your VM it should appear in the host's PulseAudio control panel as a client playing audio. Sound playing from inside the VM should work as expected.

One thing I could not solve yet is a 24 second latency when using the PulseAudio session as a recording device inside the VM. If you run e.g. Audacity and set your microphone as the recording input for the VM recording device in PulseAudio, Audacity records with a 24 second delay. This is due to a wrong handling of the buffer PulseAudio uses. The VM seems to not empty the buffer before fetching the content. A patch exists, but after looking into the Debian qemu package source the patch was not applied. There are some hints of discussion about the problem in bug trackers and mailing lists, and it seems to be fixed, but the problem has persisted until now in Debian Stretch.

Resources

Open graph icon made by Freepik from www.flaticon.com, licensed as CC 3.0 BY.