GPU PCIe passthrough on KVM

Before you start

This may look like a long post at first, In reality, it is but a few commands, the rest is output and small simple explanations, so don’t be discouraged by the length, it is really not complicated at all

Yet, you do need to check the hardware requirements before you get your hands dirty, you will find them in the “Minimum hardware requirements” section of this post

Why yet another tutorial

There are two reasons for this tutorial, Although there are many resources about every part of this hands on tutorial, they are all over the place, if you are new to Linux, you probably don’t know what to look for, The other reason is that all comprehensive tutorials assume you installed Linux with a GUI, this tutorial covers both with and without GUI, for people who don’t need the Linux GUI, going without it saves them some precious ram and disk space.

In this tutorial, I assume you do not have a GUI on your Linux machine, but I also account for those who do, So some sections of this are for those who are running GUI, for example, how to disable hibernation, which is usually already disabled if you don’t use a GUI.

The problem

I use Linux for my software development and day to day work, this means I am out of luck if i wanted to use Adobe products, or play windows games, virtual machines are nice and all, but the graphics (VNC and RDP) are prohibitively weak for those applications. And I don’t really want another computer sitting on my desk, and last but not least dual boot is a hassle and has it’s limitations (Switching between my work machine and the adobe machine for the same task for example).

I also haven’t played any games in ages (Maybe last game I can say I played regularly was starcraft !! Yes the one developed in 1996), So i am excited about finding out what games look like at this stage.

The solution

Virtual machines can still be the solution, but with dedicated GPUs and USB ! passing the whole PCIe cards to the KVM guest. this is called PCI-PASSTHROUGH. I want to pass both my Amazing top of the line GPU, the NVIDIA GTX 1650 (I’m just kidding, I know it is old and entry level)… to the guest OS which has windows, as well as my “VIA VL805/806 xHCI USB 3.0 Controller” and then should be able to run all the games my heart desires ! I basically shouldn’t be able to tell that I am working on a virtual machine.

Because this tutorial sould cover people who do not have a GUI installed, I will be setting up KVM machines with XML files, fixing networking with /network/interfaces, and so forth…

My Hardware

  • A very old ASUS P9X79 motherboard with an I7 CPU (i7-4930K) and no internal GPU, You on the other hand might have an Internal GPU with your CPU which should spare you the need to install 2 GPUs on your motherboard’s PCIe slots
  • A 4K LG monitor (Relevant for the testing phase)
  • Two graphics cards, NVIDIA GT 210 and NVIDIA GT 1650, you might only need one extrnal GPU if you have an internal GPU
  • VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller PCIe card
  • IRRELEVANT: Water cooling (I will regret this soon) because I don’t want to hear loud fans, this way 3 radiators, 2 of them passively cooled, and one with a fan should cut down the noise considerably.

Minimum hardware requirements

  • If you are trying to pass the second GPU of a laptop rather than a desktop, your laptop needs to have a Multiplexer for the HDMI port (Allowing you to pick the GPU connected to the physical HDMI connector), this is usually only available in high end and gaming laptops, so you will have to do your own research about the laptop you have.
  • A motherboard and CPU that both support IOMMU (Called VT-d on intel, AMD-Vi: AMD IOMMUv2 on AMD), It should also be enabled in your bios (UEFI)
  • A gpu that supports UEFI/IOMMU (most modern GPUs down to GT 710 do)
  • A USB PCI-e card (If you want, just a preference of mine, but you can simply redirect ports if you so chose)

So, How I intend to go about that is simple, I have a very old NVIDIA GT 210 (Which will become my host operating system’s graphics card), and pass my 1650 graphics card to a Windows Guest machine to play games and use Adobe premier. I will also be delegating a USB3 PCI-e card to the guest system (Keyboard, mouse and god knows what else)

I will assume you know nothing about virtualization and very little about Linux and take you there step by step

1- go into bios/EFI and make sure VT-d and IOMMU are turned on.

2- Don’t install the other GPU and the USB card in yet, this step is not important, but it will help you avoid having linux install unnecessary drivers for them, so let us install the OS with one GPU *(Probably built in in most cases)

3- Install Debian 12, with or without GNOME (or pick whatever GUI you like, this tutorial is GUI agnostic).

4- Physically install the GPU and USB card in the computer case

5- Let us install a few packages, I use two types of virtualization, LXC and KVM, if you don’t want LXC, simply don’t install it

apt-get install lxc ntp ntpdate debootstrap logrotate net-tools
apt-get install bridge-utils (No need, it is installed in the command above)
apt-get install libvirt-daemon-system libvirt-clients
apt-get install qemu-kvm virtinst nmap resolvconf 

6- IF YOU HAVE GUI: Disable system suspend and hibernation

If you have a GUI, odds are it is also set to suspend or hibernate after some time, so if that is the case

6.1- Disable the following systemd targets

systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

6.2- Now, if you want to make sure all is good, you need to reboot, then execute the following command

systemctl status sleep.target suspend.target hibernate.target hybrid-sleep.target

All 3 should now read “Active: inactive (dead)”

7- IF YOU HAVE GUI: Optional: Disable network manager, and enable /etc/network/interfaces

I am adding this step because this tutorial covers people with no GUI, so to make things persistent, This step makes both groups of people use the same tools, needless to say, it serves no functional purpose as network manager should do the job just fine.

7.1 – Compose a network/interfaces file

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto br0
	iface br0 inet static
	bridge_ports eno1
	bridge_fd 0
	address 192.168.7.120
	netmask 255.255.255.0
	gateway 192.168.7.1
	bridge_stp off
	bridge_maxwait 0
	dns-nameservers 8.8.8.8
	dns-nameservers 8.8.4.4

7.2- disable network manager

systemctl stop NetworkManager
systemctl disable NetworkManager

8- Check if your system supports IOMMU (Intel VT-d, and AMD IOMMU)

On the linux terminal, Let us check if our CPU supports the things we need

grep --color -E "vmx|svm" /proc/cpuinfo

If, you see vmx (Intel), or svm (AMD) you are good to go, the command above displays them in color for your convenience, In my case, because I am using an Intel CPU, I can see VMX in red which is good news

9- Update GRUB

Now, we need to modify grub ! Grub is the boot loader for my Debian machine, so need to fix it as follows, I commented the existing and added a new line for clarity, In your case, you may just want to add the missing few characters ” intel_iommu=on”

Edit /etc/default/grub

# GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUD_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

Now run

update-grub

Or

grub-mkconfig -o /boot/grub/grub.cfg

Now, reboot your machine, after the reboot, use the following command to make sure it ends with intel_iommu=on

cat /proc/cmdline

10- Inspecting our PCI cards

Now, let us take a quick look at the PCI devices i have that have the word nvidia in them

lspci -nn | grep -i nvidia

And the results of the command above were

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2)
01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)

As you can clearly see, every GPU comes with it’s own Audio Device ! in a minute you should be able to see how they are grouped together and need to be passed to the Guest OS together

11- Change the driver of the GPU to be passed to guests to vfio

This is all nice and dandy, we know plenty about our setup from the command above, but now the problem is that as soon as this machine starts up, those cards are grabbed by the drivers (nouveau by default), and that is not good, to overcome that, the graphics card that is meant to be passed to guests needs to be grabbed by the vfio driver. So let us tell that driver what we want it to grab

The card I want to assign to the Guest KVM machine is the 1650, according to the above, this card has devices, a VGA card and a sound card bearing the IDs 10de:1f82 and 10de:10fa ! to assign those a different driver, we have a few steps that need to be done

11.1- Edit the file /etc/modprobe.d/vfio.conf, and add a line similar to the following, the values in this line correspond to my own 1650 card from above, so you need to change those to match your card

options vfio-pci ids=10de:1f82,10de:10fa

11.2- We also want vfio to capture the card before any other driver, so we create the file “/etc/modprobe.d/nvidia.conf” and fill it with the following content

softdep nouveau pre: vfio-pci 
softdep nvidia pre: vfio-pci 
softdep nvidia* pre: vfio-pci

11.3- Now that vfio knows what card to pick, let us enable vfio with the following line

echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

11.4- Now, edit the file /etc/modules, since we need a few modules for what is to come, add the following lines to it

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

If your kernel was not compiled with vfio, you would have needed to run the command “modprobe vfio_pci”, but it is generally compiled in the kernel

11.6- and finally, you might want to restart the computer before the next step,

12- Check that everything is good

let us explore our IOMMU groups and PCI devices etc… again, we want to see what devices have been moved to VFIO, the following command should show you what driver is being used in the “Kernel driver in use:” section, it also gives you more information such as the board manufacturer and IOMMU group the device belongs to (Every VGA card shares a group with it’s own audio adapter for example, and they must be passed together to the guest)

So, to recap, after running the following command, the VGA card you intend to pass to the guest and it’s audio device should have vfio_pci in its “Kernel driver in use:” section

lspci -vnn

In my case, the Geforece 210 GPU and it’s audio device were group 21, while the NVIDIA GEFORCE 1650 belonged to group 22, So the lines of interest would be

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. GT218 [GeForce 210] [1043:852d]
        Flags: bus master, fast devsel, latency 0, IRQ 51, IOMMU group 21
        Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: nouveau
        Kernel modules: nouveau


01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
        Subsystem: ASUSTeK Computer Inc. High Definition Audio Controller [1043:852d]
        Flags: bus master, fast devsel, latency 0, IRQ 54, IOMMU group 21
        Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd TU117 [GeForce GTX 1650] [1458:3fcb]
        Flags: fast devsel, IRQ 11, IOMMU group 22
        Memory at f8000000 (32-bit, non-prefetchable) [disabled] [size=16M]
        Memory at a0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at b0000000 (64-bit, prefetchable) [disabled] [size=32M]
        I/O ports at d000 [disabled] [size=128]
        Expansion ROM at f9000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fcb]
        Flags: fast devsel, IRQ 10, IOMMU group 22
        Memory at f9080000 (32-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

13- Create the virtual machine, pass the PCI-e cards, and install windows.

In this example, I am making a 200GB hard drive in the location provided, assigning 12 gigabytes of ram, and starting it with a Windows installation DVD that i downloaded from microsoft’s website, and just because I can, I am leaving the VNC graphics adapter in, it simplifies my life.

virt-install –name gpu_win_adobe –os-variant win10 –ram 12288 –disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=200,sparse=yes –vcpu 6 –network bridge=br0 –hvm –graphics vnc,listen=0.0.0.0 –host-device 02:00.0 –noautoconsole –features kvm_hidden=on –cdrom /hds/12tb/Windows10_2023_11_21.iso –boot cdrom,hd

The machine should fire up the windows installer, and you can VNC to it via any VNC client of your choice, Now if you are doing this remotely from another let’s say windows machine, you can tunnel to your own machine and then VNC to it, I have written a post about doing this here

Now once the install is done, SHUT DOWN THE VIRTUAL MACHINE, and let us pass my GPU and USB cards to that machine, we do that by editing the file /etc/libvirt/quemu/gpu_win_adobe.xml , I have included the complete file that I use here for you so that you can compare and make changes, the parts that are noteworthy and relevant are the THREE <hostdev….> sections right before <memballoon model=’virtio’>, the first of which is my VGA card, the second is the sound card that comes with that VGA card, and the third being the USB controller

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit gpu_win_adobe
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>gpu_win_adobe</name>
  <uuid>968e01b8-8f35-4cd1-86b9-19839304912c</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>12</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='123456789123'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='6' threads='2'/>
    <feature policy='disable' name='hypervisor'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' discard='unmap'/>
      <source file='/hds/12tb/virts_2024/kvm/gpu_win_adobe/main.qcow2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:7c:22:50'/>
      <source bridge='br0'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Now, reflect the changes you made to the card with “virsh define /etc/libvirt/qemu/gpu_win_adobe.xml” then fire the mchine back up with “virsh start gpu_win_adobe“. sw

14- Keyboard and mouse support

Even though I do have USB on the virtual machine (Guest VM), having more than 1 keyboard and 1 mouse on my desk is a problem from both an inconvinience and a space prespective, there are many solutions out there, one of them is a cheap hardware KVM for a couple of dollars, where we only connect the keyboard and mouse, and the other is software which is much more convinient. YET, if you are using wayland, those solutions will not work, hence, I am stuck with a hardware switch.

The popular software solution is synergy, but there is an alternative solution based on synergy that is already available for linux in the debian reporitories, you can find it here (Barrier) and later (Input Leap)

12- A useful script

The folowing script looks in the IOMMU groups folder, and displays what devices are in what group, so it will come in handy at one point, so i thought i’d add it here, you can skip this section, but there is a chance you will need this functionality once you want to do other things on your own

so create a new file named check_iommu_groups.sh and add the following script to it

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Then run it like so (./check_iommu_groups.sh)

Once run you should be able to see the devices grouped together

IOMMU Group 22:
        02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
        02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
dmesg | grep -E "DMAR|IOMMU"
dmesg | grep -i vfio
dmesg |grep AMD-Vi (For amd)

update-initramfs -u
update-initramfs -u -k all

If you get the error [ 318.990400] DMAR: [DMA Read NO_PASID] Request device [02:00.0] fault addr 0x9e4b2000 [fault reason 0x06] PTE Read access is not set running dmesg | grep -E “DMAR|IOMMU”, odds are your vga card was captured by a different driver, otherwise If everything is okay, and we have not had any error or anything of the sort, we are ready to make KVM virtual machines

A virtual machine with no Passthrough first, this is my sandbox windows installation

virt-install --name windows_yazeed --os-variant win10 --ram 12288 --vcpu 6 --disk path=/hds/12tb/virts_2024/kvm/windows_yazeed/maint.qcow2,size=2000,sparse=yes --graphics vnc,listen=0.0.0.0 --noautoconsole --hvm --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd

Running it and connecting to it via VNC, done with windows installation, now another one, but this one is for the GPU passthrough

virt-install --name gpu_win_adobe --os-variant win10 --ram 12288 --disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=2,sparse=yes --vcpu 6 --network bridge=br0 --hvm --graphics vnc,listen=0.0.0.0 --host-device 02:00.0 --noautoconsole --features kvm_hidden=on --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd

But before we run this one, we will need to fix some stuff in the XML file ! /etc/libvirt/quemu/gpu_win_adobe.xml, so right before the <memballoon model=’virtio’> near the end, I added the needed parts based on my GPU’s position on the PCIe buss , I have included the full file so that you can compare my changes to the file generated by the command above

Whenever you change the xml file, you will need to tell libvirt about it, you can do that with the command

virsh define /etc/libvirt/qemu/gpu_win_adobe.xml

Hard drive power draw at startup

The maximum power draw a PC with many hard drives happens at boot time, in my case, the PC is a n intel atom D525MW, which hardly draws any power

What this means is that I need an oversized power supply that only does its thing at startup, then becomes an inefficient power supply right after, why this is particularly important is because this computer runs on a UPS, and the number of minutes it can stay up is a very important number.

The solution is to enable PUIS (Power up in standby), what this does is allow the disks not to spin as soon as they get power, but instead, spin up upon reception of a command from the controller. so in effect, the disks are spun up sequentially (In turn).

There are two ways to enable PUIS, the first is the Jumper method when there is a PUIS jumper on the back of the drive, which is very relevant to me as I am using western digital drives with this jumper, on the western digital website, they make no mentioning of PUIS on the jumpers page, but they tmake no mentioning of pins 3 and 4 ! turns out, Pins 3 and 4, when connected together with a jumper enable PUIS on the drive.

Te other way to enable PUIS is through hdparm, to check if it is already set, use the command

hdparm -I /dev/sdb | grep "Power-Up In Standby"

If your hard drive has it as a jumper option, the command above helps you explore if the jumper trick is working, otherwise, you need to enable PUIS using hdparm

Mind you, before i explain how to set this up in drives with no jumpers, some controllers are incompatible with this feature, so you will need to at least have a machine that is compatible to set them back if they don’t work

So, to enable PUIS, use the following command

sudo hdparm --yes-i-know-what-i-am-doing -s 1 /dev/sdb

And to set it back (Disable PUIS)

hdparm -s 0 /dev/sdb

For your reference, the following is what is mentioned on the wetern digital website about Jumpers, notice that they do not provide an explination of the relevant jumpers

Mounting QCOW2 (KVM/QEMU) directly

First, the tools you need

apt-get install qemu-utils

Now, enable NBD

modprobe nbd max_part=8

Once that is enabled, connect the file as a block device

qemu-nbd --connect=/dev/nbd0 /hds/usb/virts/Windows/main.qcow2

Now, the block device should appear like any other, alongside the partitions inside !

fdisk -l

On my machine, this resulted in

Disk /dev/nbd0: 95 GiB, 102005473280 bytes, 199229440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc5324c42

Device      Boot     Start       End   Sectors  Size Id Type
/dev/nbd0p1 *         2048    104447    102400   50M  7 HPFS/NTFS/exFAT
/dev/nbd0p2         104448 198138958 198034511 94.4G  7 HPFS/NTFS/exFAT
/dev/nbd0p3      198139904 199225343   1085440  530M 27 Hidden NTFS WinRE

This disk was around 40GB, but fdisk will see the number corresponding to the largest allowed size, 100GB in this case ! let us mount the drive

mount /dev/nbd0p2 /hds/loop

Now, in this case in particular, like any other block device that held the windows operating system, more often than not, you will get the message saying

The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
Could not mount read-write, trying read-only

The solution to that is simple, follow the following two steps to remedy the issue and then force mount the file by using remove_hiberfile

ntfsfix /dev/nbd0p2
mount -t ntfs-3g -o remove_hiberfile /dev/nbd0p2 /hds/loop

The result of NTFSFIX was

Mounting volume... The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
FAILED
Attempting to correct errors...
Processing $MFT and $MFTMirr...
Reading $MFT... OK
Reading $MFTMirr... OK
Comparing $MFTMirr to $MFT... OK
Processing of $MFT and $MFTMirr completed successfully.
Setting required flags on partition... OK
Going to empty the journal ($LogFile)... OK
Checking the alternate boot sector... OK
NTFS volume version is 3.1.
NTFS partition /dev/nbd0p2 was processed successfully.

And the following mount command worked as you would expect, silently

Linux badblocks cheat sheet

1- Large disks need to have their block size specified, without it, disks like my 6TB and my 8Tb hard drives will not work, badblocks will report the following error.

badblocks: Value too large for defined data type invalid end block (5860522584): must be 32-bit value

So the solution is to add the block size, like the following for example (This one is destructive)

badblocks -b 4096 -wsv /dev/sdb

It is a good idea to LOG THE BAD SECTORS (this is the command i usually use for a destructive test)

badblocks -b 4096 -o /root/badblockslog.txt -wsv /dev/sdb

Why does my SSD slow down on copying large files

So you have a budget SSD, something like the Kingston A400, and half way through a multi gigabyte file it slows down !

Coming to think about it, this SSD does not have a DRAM cache, and should be slower but more consistent, right ? after all, you can’t run out of cache when there is no cache !

The answer is NO, just because it does not have RAM cache or even actual SLC flash, does not mean it is writing casually to its MLC flash, the controller uses trickery to speed up writing and sometimes reading

Most drives with no RAM cache (Examples below) use a method called Single Level Cell mode cache, but even this name is misleading, your cheap hard drive does not have “Single Level Cell flash memory” inside of it, Instead, it utilizes it’s own MLC by writing single entries to it (One bit of a multi level cell), then re-copying it the normal way

When you are copying a large file to the disk, all the blank space at that level has been consumed, and the disk is now writing directly to the 3 dimensional MLC flash, which is, in most cases, slower than a mechanical hard drive for sequential write !

The most common of such a controller is the Phison PS3111-S11-13, it is a relatively good controller if your purpose does not require super fast SSDs, the controller has some cool features including Bad Block management (Spare flash that automatically replaces bad cells), besides standard features like S.M.A.R.T., It also supports native command queuing (NCQ), EEC error correction, so all in all, this post is not advice to stay away, this post is just hee to explain that it will be slow

Examples of such disks:

Kingston A400
– 240GB = Phison PS3111-S11-13
Silicon Power A55
– 1TB = Phison PS3111-S11-13
HIJVISION C100
– 120GB = MAS0902. (Read features below), and while the controller seems okay, at least on paper, I can not seem to be able to figure out the 64GB x 2 chips that read (TZA512G221 060422JC JWT5220364RB)

The Maxio MAS0902 SATA DRAMless controller. comparable to the (Phison PS3111) but has some interesting tech upgrades, 1- AgileECC 2 (2nd gen ECC) 2- WriteBooster 2 (2nd gen SLC write buffer), DEVSLP (low-power mode), power and thermal throttling, and end-to-end data protection. The controller also supports both TCG-Opal 2.0 and Microsoft’s eDrive (IEEE1667) full disk data encryption.

Updating the firmware on my 2TB Seagate Barracuda

Why update the firmware !

My answer here is a bit unconventional, and certainly not a fact.. I even think I am wrong, but it can’t hurt, so here it goes

Seagate recommends you update the disk’s firmware to improve performance and longevity of the hard drive, I on the other hand have an extra mission…

The firmware on a hard drive is stored partially on a chip on the PCB, and partially on the disk itself ! I know that disk platters have a data retention life of 10 years, the area where the firmware is written is never refreshed since it is read only when the disk boots up, So i am hoping (even though doubtful) that the firmware update might re-write this area of the disk and breath new life into it.

A disk certain application claims to refresh the data on that area of the disk, After testing that application I will come here and add my findings accordingly

Getting the firmware

Let us start by downloading the firmware ! To download firmware from seagate’s website, you will need to know your hard drive serial number, to do just that, open the command prompt elevated, and run the following command

wmic diskdrive get model,serialnumber

The result of me running that command is as follows

The results of executing the command above show that I have a hard drive of the model (ST2000DM001-1CH164) as you can see with a serial number that I now have (Masking serial numbers just in case seagate has a problem with me publishing them as it allows you to download the firmware using the serial number)…..

Now that I have the serial number, I can go to seagate’s firmware download page here and grab the firmware… once done, I unzip it, and the following folders appear

Creating a bootable USB flash disk

Now, inside the Bootable tools, there is a file (SeaChest_RC_2.7.4_10-18-2018.usbBootMaker.exe) that will create a bootable flash stick for you, insert a flash stick of any size that is to be deleted by this application, run the application, and now you have a bootable drive, but without the firmware, so copy the firmware folder you see above into the flash stick, and now you are ready to boot from it, for instructions how to boot from a flash stick, you will need to check with your motherboard’s manufacturer documentation, it is usually a simple thing such as hitting F11 at boot time.

Updating the firmware

Once booted, you should be presented with a linux command prompt, where we can run commands to update the firmware

To see what disks are on your system, run the following command

SeaChest_Firmware --scan

The scan should give you the handle for the drive, if you have never used linux before, the handle started with /dev/ (Short for device), and sata disks usually start with sdX (Where X starts with A and ends with a letter corresponding to the last disk you have in your system) old PATA disks usually start with hdX… but that is usually not something you need as PATA disks are virtually non existent at this stage

Now, execute the firmware update command like so

SeaChest_Firmware -d /dev/sg3 --downloadFW /firmware/filename.LOD

Now, if you want to know whether the update was successful or not, just run the scan command again, and note the firmware on it !

My Problem !

As you can see from the image below, I have 3 firmware files, named 1TB, 2TB, and 3TB, when i ran the command above, The system claimed that the update was successful, but didn’t really update the firmware, I was still stuck with 26 rather than 29 !

So i decided to use Seagates own configuration file to do the update with the command

SeaChest_Firmware -d /dev/sg3 --fwdlConfig GPCC2949.CFS

Surprise was that I got the following error

model matched but the current firmware version does not match the available updates

So, I went back in time and remembered that for this particular disk, I had changed the PCB before (Trying to get a 3TB disk to work by moving a certain chip from one board to the other, diagnosis turned out the problem is not the PCB)… So instead of flashing the 2TB firmware file, I flashed the 3TB, and what do you know, It worked.

Anyway, I will come back with screenshots of the whole thing… and more data for those who are having trouble updating their firmware, until then, hang in there

Mounting unclean NTFS windows drive in Linux

Whenever i get the following message

mount /dev/sdd1 /hds/sgt2tb
The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
Could not mount read-write, trying read-only

The command

ntfsfix /dev/sdd1

resolves the issue, and produces the following message

Mounting volume... The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
FAILED
Attempting to correct errors...
Processing $MFT and $MFTMirr...
Reading $MFT... OK
Reading $MFTMirr... OK
Comparing $MFTMirr to $MFT... OK
Processing of $MFT and $MFTMirr completed successfully.
Setting required flags on partition... OK
Going to empty the journal ($LogFile)... OK
Checking the alternate boot sector... OK
NTFS volume version is 3.1.
NTFS partition /dev/sdd1 was processed successfully

The same mount command you see here will now work flawlessly

mount /dev/sdd1 /hds/sgt2tb

I am still unsure what process from the mentioned above is responsible, as this oftentimes pops up on drives that were never system drives, so there is no hibernation file problem

Resume bad blocks where it was stopped

The answer to this should be simple, I initiated the test with

badblocks -nsv /dev/sdb

, first, interrupt bad blocks with ctrl+c, the output should be

Checking for bad blocks in non-destructive read-write mode
From block 0 to 1953514583
Checking for bad blocks (non-destructive read-write test)
Testing with random pattern:   0.92% done, 49:38 elapsed. (0/0/0 errors)
 21.32% done, 18:49:24 elapsed. (0/0/0 errors)

Interrupted at block 416437376

Interrupt caught, cleaning up

Okay, so we know what blocks it was supposed to check (1 through 1953514583), and where it was interrupted (416437376)

So i will ask it to resume testing from where it finished (-1), up to the end

badblocks -nsv /dev/sdb 1953514583 416437375

n = Non destructive
s = Show progress
v = tell us about what you find !

The new run should tell you the percentage correctly, but the time counter will be reset to zero, as it is only counting how long this run has been running for

One thing to note is that bad blocks can be used to instruct the filesystem to avoid the bad blocks, but it also allows the disk’s firmware to substitute bad blocks with spare blocks, so that the disk works again with no intervention from your end !

So for my 2TB hard drive…

416437375 = 21% (13 hours)
619014719 = 31.6% (+23:22)
627995199 = 32.15% (+1:04)
667782398 = 34.18% (+4:46)
715469885 = 36.62% (+5:44)
827834875 = 42.38%

While running the tests, you might want to keep an eye on the hard drive temperature with a command like

hddtemp /dev/sdb

To create a log file of the bad blocks, every run should have it’s own file !

badblocks -nsv -o /root/badblocks3.txt /dev/sdb 1953514583 627995198

The concatenation of those files you are creating is very useful in creating a file system if you ever decide to format the drive later !, but the recommended way is using badblocks with the other disk tools directly

while the test is running, you will see 3 numbers that correspond to readerror/writeerror/corruptionerror

LG UL550 vs the ASUS VP28UQGL

Update: the official winner in this is the ASUS, mainly for having built in speakers
Update: After buying the LG for myself and the ASUS for a friend, The ASUS does pivot 90 degrees as well !!! the only real advantage i see in the LG is the IPS display (Color accuracy for graphics designers that you will not be able to detect with the untrained eye)

Because the place where i buy my monitors currently only has 2 4K monitors in the 27″ category, I had to compare them to pick one, for most people the ASUS is the clear winner, unless you are a graphics designer, then it is the LG (IPS has better color), I will probably buy the LG because of it’s Pivot feature (UPDATE-They both have pivot), but most people would want the ASUS, in any case, here are the specs for both side by side

The reason the LG is more expensive is because it has IPS, IPS has only one advantage, color, and only to a degree that affects a graphics designer, in fact, IPS has disadvantages too compared to TN (twisted nematic).

Feature ASUS VP28UQGL lg ul550
PRICE 259 JOD ($249 at amazon) 275 JOD ($326 at amazon)
SIZE 28 (620.93 x 341.28 mm) 27
Response time (True native) TN is usually faster, but not mentioned 5ms
Response time (GTG) 1ms 1ms
Panel Type TN (twisted nematic) / LED IPS (in-plane switching)
Resolution 3840×2160 3840 x 2160
Pixel Pitch (mm) 0.160mm 0.1554 x 0.1554 mm
Refresh Rate 60Hz 60Hz
USB Video no no
USB HUB no no
HDMI 3 (2.0) but can only find 2 in manual, manual is for whole series 2 (No mentioning, but most likely 2.0a or 2.1)
DP 1 = 1 (1.2) 1 (No mentioning of Version)
HEADPHONE 1 1
BRIGHTNESS 300cd/㎡ 300cd (typ) / 240cd (Min)
CONTRAST 1000:1 (MIN/TYP) 700:1 (Min.), 1000:1 (Typ.)
COLOR 10BIT (1073.7M) – 94%sRGB IPS has better color
HDR   10
Power Consumption (Typ.) <31 36 – 41
POSITIONS Tilt : Yes (+20° ~ -5°), Pivot(Not auto) Tilt (3°)/Height (Lower and raise)/Pivot(Not auto)
VESA MOUNTING NO YES
NVIDIA/AMD Radeon FreeSync Radeon FreeSync™
SPEAKERS` YES 2W X 2 NO
Buttons Joystick Joystick
power adapter Internal (Thicker, and less heat management) external, 19v, 2a
dimensions with stand 660.4 x 672.5 x 226.3 mm 622.6 mm x 572.2 mm x 230.0 mm
dimensions without stand 660.4 x 380.8 x 62.2 mm 622.6 mm x 371.0 mm x 45.8 mm
Blue light filter Yes No
OSD Nicer and easier Does the job perfectly, but less nice

The stand is also something i like about the LG, even though it takes more space on the desk, the space is not completely taken by the stand, the half oval shape allows you to put your things on the table within the stand, it also has some height adjustment (just a bit, for the portrait pivot mode mostly so it is not so much a standing desk in any way) that allows me to raise it a bit when i need to for more comfort