GPU PCIe passthrough on KVM

Before you start

This may look like a long post at first, In reality, it is but a few commands, the rest is output and small simple explanations, so don’t be discouraged by the length, it is really not complicated at all

Yet, you do need to check the hardware requirements before you get your hands dirty, you will find them in the “Minimum hardware requirements” section of this post

Why yet another tutorial

There are two reasons for this tutorial, Although there are many resources about every part of this hands on tutorial, they are all over the place, if you are new to Linux, you probably don’t know what to look for, The other reason is that all comprehensive tutorials assume you installed Linux with a GUI, this tutorial covers both with and without GUI, for people who don’t need the Linux GUI, going without it saves them some precious ram and disk space.

In this tutorial, I assume you do not have a GUI on your Linux machine, but I also account for those who do, So some sections of this are for those who are running GUI, for example, how to disable hibernation, which is usually already disabled if you don’t use a GUI.

The problem

I use Linux for my software development and day to day work, this means I am out of luck if i wanted to use Adobe products, or play windows games, virtual machines are nice and all, but the graphics (VNC and RDP) are prohibitively weak for those applications. And I don’t really want another computer sitting on my desk, and last but not least dual boot is a hassle and has it’s limitations (Switching between my work machine and the adobe machine for the same task for example).

I also haven’t played any games in ages (Maybe last game I can say I played regularly was starcraft !! Yes the one developed in 1996), So i am excited about finding out what games look like at this stage.

The solution

Virtual machines can still be the solution, but with dedicated GPUs and USB ! passing the whole PCIe cards to the KVM guest. this is called PCI-PASSTHROUGH. I want to pass both my Amazing top of the line GPU, the NVIDIA GTX 1650 (I’m just kidding, I know it is old and entry level)… to the guest OS which has windows, as well as my “VIA VL805/806 xHCI USB 3.0 Controller” and then should be able to run all the games my heart desires ! I basically shouldn’t be able to tell that I am working on a virtual machine.

Because this tutorial sould cover people who do not have a GUI installed, I will be setting up KVM machines with XML files, fixing networking with /network/interfaces, and so forth…

My Hardware

  • A very old ASUS P9X79 motherboard with an I7 CPU (i7-4930K) and no internal GPU, You on the other hand might have an Internal GPU with your CPU which should spare you the need to install 2 GPUs on your motherboard’s PCIe slots
  • A 4K LG monitor (Relevant for the testing phase)
  • Two graphics cards, NVIDIA GT 210 and NVIDIA GT 1650, you might only need one extrnal GPU if you have an internal GPU
  • VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller PCIe card
  • IRRELEVANT: Water cooling (I will regret this soon) because I don’t want to hear loud fans, this way 3 radiators, 2 of them passively cooled, and one with a fan should cut down the noise considerably.

Minimum hardware requirements

  • If you are trying to pass the second GPU of a laptop rather than a desktop, your laptop needs to have a Multiplexer for the HDMI port (Allowing you to pick the GPU connected to the physical HDMI connector), this is usually only available in high end and gaming laptops, so you will have to do your own research about the laptop you have.
  • A motherboard and CPU that both support IOMMU (Called VT-d on intel, AMD-Vi: AMD IOMMUv2 on AMD), It should also be enabled in your bios (UEFI)
  • A gpu that supports UEFI/IOMMU (most modern GPUs down to GT 710 do)
  • A USB PCI-e card (If you want, just a preference of mine, but you can simply redirect ports if you so chose)

So, How I intend to go about that is simple, I have a very old NVIDIA GT 210 (Which will become my host operating system’s graphics card), and pass my 1650 graphics card to a Windows Guest machine to play games and use Adobe premier. I will also be delegating a USB3 PCI-e card to the guest system (Keyboard, mouse and god knows what else)

I will assume you know nothing about virtualization and very little about Linux and take you there step by step

1- go into bios/EFI and make sure VT-d and IOMMU are turned on.

2- Don’t install the other GPU and the USB card in yet, this step is not important, but it will help you avoid having linux install unnecessary drivers for them, so let us install the OS with one GPU *(Probably built in in most cases)

3- Install Debian 12, with or without GNOME (or pick whatever GUI you like, this tutorial is GUI agnostic).

4- Physically install the GPU and USB card in the computer case

5- Let us install a few packages, I use two types of virtualization, LXC and KVM, if you don’t want LXC, simply don’t install it

apt-get install lxc ntp ntpdate debootstrap logrotate net-tools
apt-get install bridge-utils (No need, it is installed in the command above)
apt-get install libvirt-daemon-system libvirt-clients
apt-get install qemu-kvm virtinst nmap resolvconf 

6- IF YOU HAVE GUI: Disable system suspend and hibernation

If you have a GUI, odds are it is also set to suspend or hibernate after some time, so if that is the case

6.1- Disable the following systemd targets

systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

6.2- Now, if you want to make sure all is good, you need to reboot, then execute the following command

systemctl status sleep.target suspend.target hibernate.target hybrid-sleep.target

All 3 should now read “Active: inactive (dead)”

7- IF YOU HAVE GUI: Optional: Disable network manager, and enable /etc/network/interfaces

I am adding this step because this tutorial covers people with no GUI, so to make things persistent, This step makes both groups of people use the same tools, needless to say, it serves no functional purpose as network manager should do the job just fine.

7.1 – Compose a network/interfaces file

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto br0
	iface br0 inet static
	bridge_ports eno1
	bridge_fd 0
	address 192.168.7.120
	netmask 255.255.255.0
	gateway 192.168.7.1
	bridge_stp off
	bridge_maxwait 0
	dns-nameservers 8.8.8.8
	dns-nameservers 8.8.4.4

7.2- disable network manager

systemctl stop NetworkManager
systemctl disable NetworkManager

8- Check if your system supports IOMMU (Intel VT-d, and AMD IOMMU)

On the linux terminal, Let us check if our CPU supports the things we need

grep --color -E "vmx|svm" /proc/cpuinfo

If, you see vmx (Intel), or svm (AMD) you are good to go, the command above displays them in color for your convenience, In my case, because I am using an Intel CPU, I can see VMX in red which is good news

9- Update GRUB

Now, we need to modify grub ! Grub is the boot loader for my Debian machine, so need to fix it as follows, I commented the existing and added a new line for clarity, In your case, you may just want to add the missing few characters ” intel_iommu=on”

Edit /etc/default/grub

# GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUD_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

Now run

update-grub

Or

grub-mkconfig -o /boot/grub/grub.cfg

Now, reboot your machine, after the reboot, use the following command to make sure it ends with intel_iommu=on

cat /proc/cmdline

10- Inspecting our PCI cards

Now, let us take a quick look at the PCI devices i have that have the word nvidia in them

lspci -nn | grep -i nvidia

And the results of the command above were

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2)
01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)

As you can clearly see, every GPU comes with it’s own Audio Device ! in a minute you should be able to see how they are grouped together and need to be passed to the Guest OS together

11- Change the driver of the GPU to be passed to guests to vfio

This is all nice and dandy, we know plenty about our setup from the command above, but now the problem is that as soon as this machine starts up, those cards are grabbed by the drivers (nouveau by default), and that is not good, to overcome that, the graphics card that is meant to be passed to guests needs to be grabbed by the vfio driver. So let us tell that driver what we want it to grab

The card I want to assign to the Guest KVM machine is the 1650, according to the above, this card has devices, a VGA card and a sound card bearing the IDs 10de:1f82 and 10de:10fa ! to assign those a different driver, we have a few steps that need to be done

11.1- Edit the file /etc/modprobe.d/vfio.conf, and add a line similar to the following, the values in this line correspond to my own 1650 card from above, so you need to change those to match your card

options vfio-pci ids=10de:1f82,10de:10fa

11.2- We also want vfio to capture the card before any other driver, so we create the file “/etc/modprobe.d/nvidia.conf” and fill it with the following content

softdep nouveau pre: vfio-pci 
softdep nvidia pre: vfio-pci 
softdep nvidia* pre: vfio-pci

11.3- Now that vfio knows what card to pick, let us enable vfio with the following line

echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

11.4- Now, edit the file /etc/modules, since we need a few modules for what is to come, add the following lines to it

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

If your kernel was not compiled with vfio, you would have needed to run the command “modprobe vfio_pci”, but it is generally compiled in the kernel

11.6- and finally, you might want to restart the computer before the next step,

12- Check that everything is good

let us explore our IOMMU groups and PCI devices etc… again, we want to see what devices have been moved to VFIO, the following command should show you what driver is being used in the “Kernel driver in use:” section, it also gives you more information such as the board manufacturer and IOMMU group the device belongs to (Every VGA card shares a group with it’s own audio adapter for example, and they must be passed together to the guest)

So, to recap, after running the following command, the VGA card you intend to pass to the guest and it’s audio device should have vfio_pci in its “Kernel driver in use:” section

lspci -vnn

In my case, the Geforece 210 GPU and it’s audio device were group 21, while the NVIDIA GEFORCE 1650 belonged to group 22, So the lines of interest would be

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. GT218 [GeForce 210] [1043:852d]
        Flags: bus master, fast devsel, latency 0, IRQ 51, IOMMU group 21
        Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: nouveau
        Kernel modules: nouveau


01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
        Subsystem: ASUSTeK Computer Inc. High Definition Audio Controller [1043:852d]
        Flags: bus master, fast devsel, latency 0, IRQ 54, IOMMU group 21
        Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd TU117 [GeForce GTX 1650] [1458:3fcb]
        Flags: fast devsel, IRQ 11, IOMMU group 22
        Memory at f8000000 (32-bit, non-prefetchable) [disabled] [size=16M]
        Memory at a0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at b0000000 (64-bit, prefetchable) [disabled] [size=32M]
        I/O ports at d000 [disabled] [size=128]
        Expansion ROM at f9000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fcb]
        Flags: fast devsel, IRQ 10, IOMMU group 22
        Memory at f9080000 (32-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

13- Create the virtual machine, pass the PCI-e cards, and install windows.

In this example, I am making a 200GB hard drive in the location provided, assigning 12 gigabytes of ram, and starting it with a Windows installation DVD that i downloaded from microsoft’s website, and just because I can, I am leaving the VNC graphics adapter in, it simplifies my life.

virt-install –name gpu_win_adobe –os-variant win10 –ram 12288 –disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=200,sparse=yes –vcpu 6 –network bridge=br0 –hvm –graphics vnc,listen=0.0.0.0 –host-device 02:00.0 –noautoconsole –features kvm_hidden=on –cdrom /hds/12tb/Windows10_2023_11_21.iso –boot cdrom,hd

The machine should fire up the windows installer, and you can VNC to it via any VNC client of your choice, Now if you are doing this remotely from another let’s say windows machine, you can tunnel to your own machine and then VNC to it, I have written a post about doing this here

Now once the install is done, SHUT DOWN THE VIRTUAL MACHINE, and let us pass my GPU and USB cards to that machine, we do that by editing the file /etc/libvirt/quemu/gpu_win_adobe.xml , I have included the complete file that I use here for you so that you can compare and make changes, the parts that are noteworthy and relevant are the THREE <hostdev….> sections right before <memballoon model=’virtio’>, the first of which is my VGA card, the second is the sound card that comes with that VGA card, and the third being the USB controller

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit gpu_win_adobe
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>gpu_win_adobe</name>
  <uuid>968e01b8-8f35-4cd1-86b9-19839304912c</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>12</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='123456789123'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='6' threads='2'/>
    <feature policy='disable' name='hypervisor'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' discard='unmap'/>
      <source file='/hds/12tb/virts_2024/kvm/gpu_win_adobe/main.qcow2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:7c:22:50'/>
      <source bridge='br0'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Now, reflect the changes you made to the card with “virsh define /etc/libvirt/qemu/gpu_win_adobe.xml” then fire the mchine back up with “virsh start gpu_win_adobe“. sw

14- Keyboard and mouse support

Even though I do have USB on the virtual machine (Guest VM), having more than 1 keyboard and 1 mouse on my desk is a problem from both an inconvinience and a space prespective, there are many solutions out there, one of them is a cheap hardware KVM for a couple of dollars, where we only connect the keyboard and mouse, and the other is software which is much more convinient. YET, if you are using wayland, those solutions will not work, hence, I am stuck with a hardware switch.

The popular software solution is synergy, but there is an alternative solution based on synergy that is already available for linux in the debian reporitories, you can find it here (Barrier) and later (Input Leap)

12- A useful script

The folowing script looks in the IOMMU groups folder, and displays what devices are in what group, so it will come in handy at one point, so i thought i’d add it here, you can skip this section, but there is a chance you will need this functionality once you want to do other things on your own

so create a new file named check_iommu_groups.sh and add the following script to it

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Then run it like so (./check_iommu_groups.sh)

Once run you should be able to see the devices grouped together

IOMMU Group 22:
        02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
        02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
dmesg | grep -E "DMAR|IOMMU"
dmesg | grep -i vfio
dmesg |grep AMD-Vi (For amd)

update-initramfs -u
update-initramfs -u -k all

If you get the error [ 318.990400] DMAR: [DMA Read NO_PASID] Request device [02:00.0] fault addr 0x9e4b2000 [fault reason 0x06] PTE Read access is not set running dmesg | grep -E “DMAR|IOMMU”, odds are your vga card was captured by a different driver, otherwise If everything is okay, and we have not had any error or anything of the sort, we are ready to make KVM virtual machines

A virtual machine with no Passthrough first, this is my sandbox windows installation

virt-install --name windows_yazeed --os-variant win10 --ram 12288 --vcpu 6 --disk path=/hds/12tb/virts_2024/kvm/windows_yazeed/maint.qcow2,size=2000,sparse=yes --graphics vnc,listen=0.0.0.0 --noautoconsole --hvm --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd

Running it and connecting to it via VNC, done with windows installation, now another one, but this one is for the GPU passthrough

virt-install --name gpu_win_adobe --os-variant win10 --ram 12288 --disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=2,sparse=yes --vcpu 6 --network bridge=br0 --hvm --graphics vnc,listen=0.0.0.0 --host-device 02:00.0 --noautoconsole --features kvm_hidden=on --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd

But before we run this one, we will need to fix some stuff in the XML file ! /etc/libvirt/quemu/gpu_win_adobe.xml, so right before the <memballoon model=’virtio’> near the end, I added the needed parts based on my GPU’s position on the PCIe buss , I have included the full file so that you can compare my changes to the file generated by the command above

Whenever you change the xml file, you will need to tell libvirt about it, you can do that with the command

virsh define /etc/libvirt/qemu/gpu_win_adobe.xml

Mounting QCOW2 (KVM/QEMU) directly

First, the tools you need

apt-get install qemu-utils

Now, enable NBD

modprobe nbd max_part=8

Once that is enabled, connect the file as a block device

qemu-nbd --connect=/dev/nbd0 /hds/usb/virts/Windows/main.qcow2

Now, the block device should appear like any other, alongside the partitions inside !

fdisk -l

On my machine, this resulted in

Disk /dev/nbd0: 95 GiB, 102005473280 bytes, 199229440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc5324c42

Device      Boot     Start       End   Sectors  Size Id Type
/dev/nbd0p1 *         2048    104447    102400   50M  7 HPFS/NTFS/exFAT
/dev/nbd0p2         104448 198138958 198034511 94.4G  7 HPFS/NTFS/exFAT
/dev/nbd0p3      198139904 199225343   1085440  530M 27 Hidden NTFS WinRE

This disk was around 40GB, but fdisk will see the number corresponding to the largest allowed size, 100GB in this case ! let us mount the drive

mount /dev/nbd0p2 /hds/loop

Now, in this case in particular, like any other block device that held the windows operating system, more often than not, you will get the message saying

The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
Could not mount read-write, trying read-only

The solution to that is simple, follow the following two steps to remedy the issue and then force mount the file by using remove_hiberfile

ntfsfix /dev/nbd0p2
mount -t ntfs-3g -o remove_hiberfile /dev/nbd0p2 /hds/loop

The result of NTFSFIX was

Mounting volume... The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
FAILED
Attempting to correct errors...
Processing $MFT and $MFTMirr...
Reading $MFT... OK
Reading $MFTMirr... OK
Comparing $MFTMirr to $MFT... OK
Processing of $MFT and $MFTMirr completed successfully.
Setting required flags on partition... OK
Going to empty the journal ($LogFile)... OK
Checking the alternate boot sector... OK
NTFS volume version is 3.1.
NTFS partition /dev/nbd0p2 was processed successfully.

And the following mount command worked as you would expect, silently

Nested virtualization in KVM

The reason I am enabling this in my virtual machine is to develop with android studio under windows or Linux in a dedicated development machine (Let us call it an android development virtual machine), you will need to enable nested virtualization for the virtual android phone that comes with Android studio, there are many occasions where you need nested virtualization, so let us see what we need to do.

1- Check if our system allows nested virtualization with the following line

cat /sys/module/kvm_intel/parameters/nested 

If this returns a Y or a 1, then we are good to go to the next step, if not, then execute the following to enable the feature on the host system

echo 'options kvm_intel nested=1' >> /etc/modprobe.d/qemu-system-x86.conf 

Now, with that out of the way, we can move to the next step

2- Enable nested virtualization in the config of the virtual machine, either with virsh edit or edit the file manually and reload it, whatever you are used to doing should work

virsh edit androiddev

Now, specify either host-model OR host-passthrough, host model is more compatible when moving the virtual machine to a new CPU, while host-passthrough will deliver absolutly all CPU features to the guest os, but is very unfriendly to moving the machine to a different KVM host

<cpu mode='host-model'> 

Installing MacOS in a virtual machine (KVM) under linux

This is a simple task, and it is only simple because of foxlet (@FoxletFox on twitter)

Anyway, let us get to setting it up, to begin with, you don’t need to download MacOS, when using foxlet’s macOS-Simple-KVM, your virtual machine downloads MacOS on it’s own

Step 1: Make sure you have KVM ! and the relevant tools

apt-get install qemu-kvm libvirt-daemon qemu-system qemu-utils python3 python3-pip bridge-utils virtinst libvirt-daemon-system virt-manager

You know, the usual kvm setup ;), I am hoping you already have KVM, if not, see this post and install KVM first

Now that you have kvm, you need to insure that vhost_net is installed, loaded and enabled

modprobe vhost_net
lsmod | grep vhost

You will also need git to download macOS-Simple-KVM

git
git clone https://github.com/foxlet/macOS-Simple-KVM.git

Now, download MacOS base image that will download the rest of the operating system (catalina is the latest ?) options in that script are –high-sierra, –mojave, or –catalina.

./jumpstart.sh --catalina

Connecting to Windows KVM with VNC and putty tunnel

The setup assumed in this post is as follows, you are working on a remote windows computer, there is a Linux KVM host computer running guest virtual machines somewhere (OS of guest irrelevant), and you would like to connect to a guest machine’s console (which may be running windows, linux, macOS, or any operating system)

KVM, by default only allows people to connect through VNC to the console of a virtual machine if they are using the local host computer, so here are the tips on creating a tunnel to the host computer and connecting to your KVM virtual machine.

Windows does not support VNC very well, (Most VNC servers don’t run well on windows), but the VNC server here is not windows, it is KVM that is providing the VNC server to the guest’s console.

1- Create a tunnel (Putty on windows), simply put, save the connection in putty to that host machine, then under tunnels you will need to have something like this (And go back and hot save again)

Just create a tunnel for port 5900 and the destination localhost:5900 (5901 for the second virtual machine and so on), leave all other tunnel options unchecked/default

2- to know which ones are enabled on your machine run this command

netstat -tlpn | grep 590

3- VNC should now connect to localhost:9500 for example (I am using tightVNC on windows), and that connection should be automatically router to the KVM host, which will display the guest’s console depending on the port (every guest has it’s own port)

gigabit Ethernet VirtIO driver for Windows 10 64bit

By default, KVM gives your virtual machine a realtek rtl8139 Ethernet adapter, with an ancient 100Mbit/Second speed, we all need gigabit Ethernet adapter for the KVM guest.

The answer is changing the string rtl8139 with virtio in the XML file of the virtual machine, then installing the drivers

The steps i use are

Run virtual machine with the realtek adapter to download the other adapter’s driver
once the adapter is there, shutdown the virtual machine guest (Windows guest), then edit the xml of the guest, and restart libvirtd
start the KVM guest again
open with VNC, start the device manager, install the driver you downloaded.

You are good, the adapter should report the speed of 10Gbit/second (10 gigabit per second)

One annoying thing is that all windows drivers come in a big ISO file, you probably just want the driver you need.

I will add the download links in the coming few days, but you can get them right now if you like from fedora, the fedora windows guest drivers should work on any linux distribution (Debian, ubuntu, etc…)