GPU PCIe passthrough on KVM

Before you start

This may look like a long post at first, In reality, it is but a few commands, the rest is output and small simple explanations, so don’t be discouraged by the length, it is really not complicated at all

Yet, you do need to check the hardware requirements before you get your hands dirty, you will find them in the “Minimum hardware requirements” section of this post

Why yet another tutorial

There are two reasons for this tutorial, Although there are many resources about every part of this hands on tutorial, they are all over the place, if you are new to Linux, you probably don’t know what to look for, The other reason is that all comprehensive tutorials assume you installed Linux with a GUI, this tutorial covers both with and without GUI, for people who don’t need the Linux GUI, going without it saves them some precious ram and disk space.

In this tutorial, I assume you do not have a GUI on your Linux machine, but I also account for those who do, So some sections of this are for those who are running GUI, for example, how to disable hibernation, which is usually already disabled if you don’t use a GUI.

The problem

I use Linux for my software development and day to day work, this means I am out of luck if i wanted to use Adobe products, or play windows games, virtual machines are nice and all, but the graphics (VNC and RDP) are prohibitively weak for those applications. And I don’t really want another computer sitting on my desk, and last but not least dual boot is a hassle and has it’s limitations (Switching between my work machine and the adobe machine for the same task for example).

I also haven’t played any games in ages (Maybe last game I can say I played regularly was starcraft !! Yes the one developed in 1996), So i am excited about finding out what games look like at this stage.

The solution

Virtual machines can still be the solution, but with dedicated GPUs and USB ! passing the whole PCIe cards to the KVM guest. this is called PCI-PASSTHROUGH. I want to pass both my Amazing top of the line GPU, the NVIDIA GTX 1650 (I’m just kidding, I know it is old and entry level)… to the guest OS which has windows, as well as my “VIA VL805/806 xHCI USB 3.0 Controller” and then should be able to run all the games my heart desires ! I basically shouldn’t be able to tell that I am working on a virtual machine.

Because this tutorial sould cover people who do not have a GUI installed, I will be setting up KVM machines with XML files, fixing networking with /network/interfaces, and so forth…

My Hardware

  • A very old ASUS P9X79 motherboard with an I7 CPU (i7-4930K) and no internal GPU, You on the other hand might have an Internal GPU with your CPU which should spare you the need to install 2 GPUs on your motherboard’s PCIe slots
  • A 4K LG monitor (Relevant for the testing phase)
  • Two graphics cards, NVIDIA GT 210 and NVIDIA GT 1650, you might only need one extrnal GPU if you have an internal GPU
  • VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller PCIe card
  • IRRELEVANT: Water cooling (I will regret this soon) because I don’t want to hear loud fans, this way 3 radiators, 2 of them passively cooled, and one with a fan should cut down the noise considerably.

Minimum hardware requirements

  • If you are trying to pass the second GPU of a laptop rather than a desktop, your laptop needs to have a Multiplexer for the HDMI port (Allowing you to pick the GPU connected to the physical HDMI connector), this is usually only available in high end and gaming laptops, so you will have to do your own research about the laptop you have.
  • A motherboard and CPU that both support IOMMU (Called VT-d on intel, AMD-Vi: AMD IOMMUv2 on AMD), It should also be enabled in your bios (UEFI)
  • A gpu that supports UEFI/IOMMU (most modern GPUs down to GT 710 do)
  • A USB PCI-e card (If you want, just a preference of mine, but you can simply redirect ports if you so chose)

So, How I intend to go about that is simple, I have a very old NVIDIA GT 210 (Which will become my host operating system’s graphics card), and pass my 1650 graphics card to a Windows Guest machine to play games and use Adobe premier. I will also be delegating a USB3 PCI-e card to the guest system (Keyboard, mouse and god knows what else)

I will assume you know nothing about virtualization and very little about Linux and take you there step by step

1- go into bios/EFI and make sure VT-d and IOMMU are turned on.

2- Don’t install the other GPU and the USB card in yet, this step is not important, but it will help you avoid having linux install unnecessary drivers for them, so let us install the OS with one GPU *(Probably built in in most cases)

3- Install Debian 12, with or without GNOME (or pick whatever GUI you like, this tutorial is GUI agnostic).

4- Physically install the GPU and USB card in the computer case

5- Let us install a few packages, I use two types of virtualization, LXC and KVM, if you don’t want LXC, simply don’t install it

apt-get install lxc ntp ntpdate debootstrap logrotate net-tools
apt-get install bridge-utils (No need, it is installed in the command above)
apt-get install libvirt-daemon-system libvirt-clients
apt-get install qemu-kvm virtinst nmap resolvconf 

6- IF YOU HAVE GUI: Disable system suspend and hibernation

If you have a GUI, odds are it is also set to suspend or hibernate after some time, so if that is the case

6.1- Disable the following systemd targets

systemctl mask

6.2- Now, if you want to make sure all is good, you need to reboot, then execute the following command

systemctl status

All 3 should now read “Active: inactive (dead)”

7- IF YOU HAVE GUI: Optional: Disable network manager, and enable /etc/network/interfaces

I am adding this step because this tutorial covers people with no GUI, so to make things persistent, This step makes both groups of people use the same tools, needless to say, it serves no functional purpose as network manager should do the job just fine.

7.1 – Compose a network/interfaces file

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto br0
	iface br0 inet static
	bridge_ports eno1
	bridge_fd 0
	bridge_stp off
	bridge_maxwait 0

7.2- disable network manager

systemctl stop NetworkManager
systemctl disable NetworkManager

8- Check if your system supports IOMMU (Intel VT-d, and AMD IOMMU)

On the linux terminal, Let us check if our CPU supports the things we need

grep --color -E "vmx|svm" /proc/cpuinfo

If, you see vmx (Intel), or svm (AMD) you are good to go, the command above displays them in color for your convenience, In my case, because I am using an Intel CPU, I can see VMX in red which is good news

9- Update GRUB

Now, we need to modify grub ! Grub is the boot loader for my Debian machine, so need to fix it as follows, I commented the existing and added a new line for clarity, In your case, you may just want to add the missing few characters ” intel_iommu=on”

Edit /etc/default/grub

GRUD_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

Now run



grub-mkconfig -o /boot/grub/grub.cfg

Now, reboot your machine, after the reboot, use the following command to make sure it ends with intel_iommu=on

cat /proc/cmdline

10- Inspecting our PCI cards

Now, let us take a quick look at the PCI devices i have that have the word nvidia in them

lspci -nn | grep -i nvidia

And the results of the command above were

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2)
01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)

As you can clearly see, every GPU comes with it’s own Audio Device ! in a minute you should be able to see how they are grouped together and need to be passed to the Guest OS together

11- Change the driver of the GPU to be passed to guests to vfio

This is all nice and dandy, we know plenty about our setup from the command above, but now the problem is that as soon as this machine starts up, those cards are grabbed by the drivers (nouveau by default), and that is not good, to overcome that, the graphics card that is meant to be passed to guests needs to be grabbed by the vfio driver. So let us tell that driver what we want it to grab

The card I want to assign to the Guest KVM machine is the 1650, according to the above, this card has devices, a VGA card and a sound card bearing the IDs 10de:1f82 and 10de:10fa ! to assign those a different driver, we have a few steps that need to be done

11.1- Edit the file /etc/modprobe.d/vfio.conf, and add a line similar to the following, the values in this line correspond to my own 1650 card from above, so you need to change those to match your card

options vfio-pci ids=10de:1f82,10de:10fa

11.2- We also want vfio to capture the card before any other driver, so we create the file “/etc/modprobe.d/nvidia.conf” and fill it with the following content

softdep nouveau pre: vfio-pci 
softdep nvidia pre: vfio-pci 
softdep nvidia* pre: vfio-pci

11.3- Now that vfio knows what card to pick, let us enable vfio with the following line

echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

11.4- Now, edit the file /etc/modules, since we need a few modules for what is to come, add the following lines to it

# /etc/modules: kernel modules to load at boot time.
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

If your kernel was not compiled with vfio, you would have needed to run the command “modprobe vfio_pci”, but it is generally compiled in the kernel

11.6- and finally, you might want to restart the computer before the next step,

12- Check that everything is good

let us explore our IOMMU groups and PCI devices etc… again, we want to see what devices have been moved to VFIO, the following command should show you what driver is being used in the “Kernel driver in use:” section, it also gives you more information such as the board manufacturer and IOMMU group the device belongs to (Every VGA card shares a group with it’s own audio adapter for example, and they must be passed together to the guest)

So, to recap, after running the following command, the VGA card you intend to pass to the guest and it’s audio device should have vfio_pci in its “Kernel driver in use:” section

lspci -vnn

In my case, the Geforece 210 GPU and it’s audio device were group 21, while the NVIDIA GEFORCE 1650 belonged to group 22, So the lines of interest would be

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. GT218 [GeForce 210] [1043:852d]
        Flags: bus master, fast devsel, latency 0, IRQ 51, IOMMU group 21
        Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: nouveau
        Kernel modules: nouveau

01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
        Subsystem: ASUSTeK Computer Inc. High Definition Audio Controller [1043:852d]
        Flags: bus master, fast devsel, latency 0, IRQ 54, IOMMU group 21
        Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd TU117 [GeForce GTX 1650] [1458:3fcb]
        Flags: fast devsel, IRQ 11, IOMMU group 22
        Memory at f8000000 (32-bit, non-prefetchable) [disabled] [size=16M]
        Memory at a0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at b0000000 (64-bit, prefetchable) [disabled] [size=32M]
        I/O ports at d000 [disabled] [size=128]
        Expansion ROM at f9000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fcb]
        Flags: fast devsel, IRQ 10, IOMMU group 22
        Memory at f9080000 (32-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

13- Create the virtual machine, pass the PCI-e cards, and install windows.

In this example, I am making a 200GB hard drive in the location provided, assigning 12 gigabytes of ram, and starting it with a Windows installation DVD that i downloaded from microsoft’s website, and just because I can, I am leaving the VNC graphics adapter in, it simplifies my life.

virt-install –name gpu_win_adobe –os-variant win10 –ram 12288 –disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=200,sparse=yes –vcpu 6 –network bridge=br0 –hvm –graphics vnc,listen= –host-device 02:00.0 –noautoconsole –features kvm_hidden=on –cdrom /hds/12tb/Windows10_2023_11_21.iso –boot cdrom,hd

The machine should fire up the windows installer, and you can VNC to it via any VNC client of your choice, Now if you are doing this remotely from another let’s say windows machine, you can tunnel to your own machine and then VNC to it, I have written a post about doing this here

Now once the install is done, SHUT DOWN THE VIRTUAL MACHINE, and let us pass my GPU and USB cards to that machine, we do that by editing the file /etc/libvirt/quemu/gpu_win_adobe.xml , I have included the complete file that I use here for you so that you can compare and make changes, the parts that are noteworthy and relevant are the THREE <hostdev….> sections right before <memballoon model=’virtio’>, the first of which is my VGA card, the second is the sound card that comes with that VGA card, and the third being the USB controller

OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit gpu_win_adobe
or other application using the libvirt API.

<domain type='kvm'>
    <libosinfo:libosinfo xmlns:libosinfo="">
      <libosinfo:os id=""/>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>12</vcpu>
    <type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='123456789123'/>
      <hidden state='on'/>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='6' threads='2'/>
    <feature policy='disable' name='hypervisor'/>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' discard='unmap'/>
      <source file='/hds/12tb/virts_2024/kvm/gpu_win_adobe/main.qcow2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    <interface type='bridge'>
      <mac address='52:54:00:7c:22:50'/>
      <source bridge='br0'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
    <console type='pty'>
      <target type='serial' port='0'/>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen=''>
      <listen type='address' address=''/>
    <audio id='1' type='none'/>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>

Now, reflect the changes you made to the card with “virsh define /etc/libvirt/qemu/gpu_win_adobe.xml” then fire the mchine back up with “virsh start gpu_win_adobe“. sw

14- Keyboard and mouse support

Even though I do have USB on the virtual machine (Guest VM), having more than 1 keyboard and 1 mouse on my desk is a problem from both an inconvinience and a space prespective, there are many solutions out there, one of them is a cheap hardware KVM for a couple of dollars, where we only connect the keyboard and mouse, and the other is software which is much more convinient. YET, if you are using wayland, those solutions will not work, hence, I am stuck with a hardware switch.

The popular software solution is synergy, but there is an alternative solution based on synergy that is already available for linux in the debian reporitories, you can find it here (Barrier) and later (Input Leap)

12- A useful script

The folowing script looks in the IOMMU groups folder, and displays what devices are in what group, so it will come in handy at one point, so i thought i’d add it here, you can skip this section, but there is a chance you will need this functionality once you want to do other things on your own

so create a new file named and add the following script to it

shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"

Then run it like so (./

Once run you should be able to see the devices grouped together

IOMMU Group 22:
        02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
        02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
dmesg | grep -E "DMAR|IOMMU"
dmesg | grep -i vfio
dmesg |grep AMD-Vi (For amd)

update-initramfs -u
update-initramfs -u -k all

If you get the error [ 318.990400] DMAR: [DMA Read NO_PASID] Request device [02:00.0] fault addr 0x9e4b2000 [fault reason 0x06] PTE Read access is not set running dmesg | grep -E “DMAR|IOMMU”, odds are your vga card was captured by a different driver, otherwise If everything is okay, and we have not had any error or anything of the sort, we are ready to make KVM virtual machines

A virtual machine with no Passthrough first, this is my sandbox windows installation

virt-install --name windows_yazeed --os-variant win10 --ram 12288 --vcpu 6 --disk path=/hds/12tb/virts_2024/kvm/windows_yazeed/maint.qcow2,size=2000,sparse=yes --graphics vnc,listen= --noautoconsole --hvm --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd

Running it and connecting to it via VNC, done with windows installation, now another one, but this one is for the GPU passthrough

virt-install --name gpu_win_adobe --os-variant win10 --ram 12288 --disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=2,sparse=yes --vcpu 6 --network bridge=br0 --hvm --graphics vnc,listen= --host-device 02:00.0 --noautoconsole --features kvm_hidden=on --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd

But before we run this one, we will need to fix some stuff in the XML file ! /etc/libvirt/quemu/gpu_win_adobe.xml, so right before the <memballoon model=’virtio’> near the end, I added the needed parts based on my GPU’s position on the PCIe buss , I have included the full file so that you can compare my changes to the file generated by the command above

Whenever you change the xml file, you will need to tell libvirt about it, you can do that with the command

virsh define /etc/libvirt/qemu/gpu_win_adobe.xml

Mounting QCOW2 (KVM/QEMU) directly

First, the tools you need

apt-get install qemu-utils

Now, enable NBD

modprobe nbd max_part=8

Once that is enabled, connect the file as a block device

qemu-nbd --connect=/dev/nbd0 /hds/usb/virts/Windows/main.qcow2

Now, the block device should appear like any other, alongside the partitions inside !

fdisk -l

On my machine, this resulted in

Disk /dev/nbd0: 95 GiB, 102005473280 bytes, 199229440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc5324c42

Device      Boot     Start       End   Sectors  Size Id Type
/dev/nbd0p1 *         2048    104447    102400   50M  7 HPFS/NTFS/exFAT
/dev/nbd0p2         104448 198138958 198034511 94.4G  7 HPFS/NTFS/exFAT
/dev/nbd0p3      198139904 199225343   1085440  530M 27 Hidden NTFS WinRE

This disk was around 40GB, but fdisk will see the number corresponding to the largest allowed size, 100GB in this case ! let us mount the drive

mount /dev/nbd0p2 /hds/loop

Now, in this case in particular, like any other block device that held the windows operating system, more often than not, you will get the message saying

The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
Could not mount read-write, trying read-only

The solution to that is simple, follow the following two steps to remedy the issue and then force mount the file by using remove_hiberfile

ntfsfix /dev/nbd0p2
mount -t ntfs-3g -o remove_hiberfile /dev/nbd0p2 /hds/loop

The result of NTFSFIX was

Mounting volume... The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Attempting to correct errors...
Processing $MFT and $MFTMirr...
Reading $MFT... OK
Reading $MFTMirr... OK
Comparing $MFTMirr to $MFT... OK
Processing of $MFT and $MFTMirr completed successfully.
Setting required flags on partition... OK
Going to empty the journal ($LogFile)... OK
Checking the alternate boot sector... OK
NTFS volume version is 3.1.
NTFS partition /dev/nbd0p2 was processed successfully.

And the following mount command worked as you would expect, silently

Step by step Unprivileged containers on Debian Bookworm

The full version of this, with an explanation of everything is here, this one is written for copy-paste and speed.

This version is meant to create unprivileged LXC containers owned by root subordinates, which in my opinion provides the best balance of security and flexibility.

  • Install Debian 12 (bookworm) on a computer or virtual machine or what have you.
  • I personally enable root access under SSH, so all the commands you see here are run as root, you may use another user with sudo if you wish, but i execute as root
  • Execute the following to install LXC (I am installing LXC and KVM) but you might want to remove KVM
apt-get update

apt-get install bridge-utils lxc libvirt-clients libvirt-daemon-system debootstrap qemu-kvm bridge-utils virtinst nmap resolvconf iotop net-tools

Most installations will have 2 users, root and another username you chose while installing the operating system,

Unprivileged containers made simple on Debian 12 (Bookworm)

IMPORTANT NOTE: This is the full version, if you just want to come in, copy some commands, and end up making unprivileged containers under root, THERE IS A SEPARATE POST FOR THAT HERE.

0- Intro

Don’t let the length fool you, I am trying to make this the simplest and fastest yet most comprehensive tutorial to having LXC (both privileged and unprivileged) up and running on debian bookworm !

I sent a previous version of this to a friend to spare myself the need to explain to him what to do, and he found the tutorial confusing ! instead of the old arrangement, having colors to denote what lines are for what task, I have decided to SEPARATE THIS INTO PARTS….

  1. Intro – About this post (You are already in it)
  2. LXC info
  3. Shared system setup (Privileged and unprivileged)
  4. Privilaged LXC step by step
  5. Shared setup for unprivileged containers
  6. Unprivileged LXC run by new user, step by step
  7. Unprivileged LXC run by root user, step by step

I hope this clears things up, the color codes will still exist, mostly because I have already done the work !

Why yet another tutorial ?

Most of the tutorials online focus on creating an extra user to use with LXC, that is one way to do it with a few drawbacks, the other way is to create a range of subordinate IDs for the root user, the advantages of this way of doing it are related to “Autostart” and filesystem sharing between host and guest.

As per usual, the primary goal of every post on this blog is my own reference, the internet is full of misleading and inaccurate stuff, and when i come back to a similar situation, I don’t want to do the research all over again.

Part 1: About LXC

Privileged VS unprivileged

Privileged containers are generally unsafe, the only advantage of privileged containers is that is is very easy to setup.

Privileged containers share the same root user with the host, so if the container root user gets compromised, the attacker can sneak into the host system, hence, unprivileged is more secure but involves some work initially to setup

What is the problem with Privileged containers

It is relatively easy to deploy LXC (Which also happens to be what is powering LXD)… You install it, run a command to create a container, and voila, a whole new Linux system within your host Linux system sharing the same kernel as the host… But there is one caveat, if a malicious user/application compromises your container, he/she would have also compromised the host machine automatically, how, the root user on both is the same user !

The solution, unprivileged containers

In comes Unprivileged containers, in this setup, we simply either map a User ID to root within the container, or, still use root, but through subordinate IDs, so instead of having the Host’s user id for root (Usually Zero) being also root inside the container, we create a user outside the container (Or a subordinate ID of root), and instruct the kernel to map this user’s ID and treat it as ID zero inside the container, So if a malicious user gets access to the container and ends up breaking out of the container, they will find themselves logged on as a different user, with privileges very close to the privileges of the user nobody, or in other words, barely any privileges

Relevant topic: User namespaces

A relevant topic to Unprivileged LXC containers is User namespaces (Starting kernel 3.8), namespaces are created with the functions clone() or unshare().

nuff with the theory, What do i need to do ?

You setup LXC, then depending on the type of container and user you need, you may want to setup Linux kernel to use that user as root in the container, but to make that happen, you will need to take a few steps to give that user the required privileges and nothing more than what is required, nothing complicated about those steps either. So let us get started

2- Shared system setup

Before writing this tutorial, I installed a copy of bookworm, enabled SSH, and got to work doing the steps you see below, the steps in this section are the same whether you plan to create privileged or unprivileged containers or both

Step 2-1: Install everything

apt-get update
apt-get install bridge-utils lxc libvirt-clients libvirt-daemon-system debootstrap qemu-kvm virtinst nmap resolvconf iotop net-tools

Step 2-2: Enable IP forwarding

Next, we need to enable IPv4 forwarding by un-commenting a line in sysctl.conf then run sysctl -p, so open sysctl.conf in your favorite linux compatible editor, and uncomment the line


Now run the following command for the effects to take place

sysctl -p

Step 2-3: Host Networking

Before creating any containers, we need to make sure the host can bridge the network to them, in Debian, this is done by editing the file /etc/network/interfaces, there are a few ways to connect the containers, your host can become a DHCP server, or you can connect the containers directly to your router

In this setup below, I am connecting the containers directly to the router.. The host machine will have the IP, IF YOU ARE USING HYPER-V, YOU WILL NEED TO ENABLE “MAC address spoofing” IN THE HYPER-V VM SETTINGS

auto br0
	iface br0 inet static
	bridge_ports eno1
	bridge_fd 0
	bridge_stp off
	bridge_maxwait 0

3- Privilaged LXC

To clarify, making a privileged container does not stop you from making unprivileged containers later, BUT, the unprivileged containers need to be different containers 😉 so you might make a privileged one, then replace it with an unprivileged one

Step 3-1: Download container

The following step is all about downloading your LXC container template ! I chose the mirror with the lowest ping time from me, but you can omit the mirror line altogether

MIRROR= lxc-create --name vm142 --template download -- --dist debian --release bookworm --arch amd64

Something unexpected happened while i was doing this, I received an error about a problem downloading, by coincidence, i rebooted the machine and it worked, my theory is that the reboot was irrelevant but if this happens to you, tell me your conclusions in the comments.

"../src/lxc/lxccontainer.c: create_run_template: 1628 Failed to create container from template"

Right after, you have a brand new LXC container which is unfortunately privileged, you can have it listed with the command “lxc-ls -f” where the f stands for fancy 😉

lxc-ls -f

Step 3-2: Edit virtual machine config

This container might not be able to start though, some editing of the config file may be necessary !

Here is this machines config file, mind the comments, this is meant to be modified to fit your networking setup, so you will need to change the IP address and relevant network address information, the machine name and rootfs path, etc…

#this is a modified LXC container config file
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist debian --release bookworm --arch amd64
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = linux64

# Container specific configuration
lxc.apparmor.profile = generated
#nesting is for having docker and other similar containerization tech inside the container, dissable it if you don't want such virtual machines in the virtual machine
lxc.apparmor.allow_nesting = 1
lxc.rootfs.path = dir:/var/lib/lxc/vm142/rootfs = vm142

# Initial Network configuration, disabled... = veth = lxcbr0 = up

#the above config was dissabled, so net.0 altogether is better left empty = empty

#Now, add networking = veth = up = br0 = eth0 = =

#App armor profile for this PRIVILEGED container
lxc.apparmor.profile = generated

#If you want this container to start with the host, uncomment the following = 1
#lxc-start.delay = 10
# #the order, the higher the earlier ;) 
#lxc.start.order = 30

# Container specific configuration (Not initially there)
lxc.tty.max = 4
lxc.pty.max = 1024

Problem : One remaining problem was that the virtual machine was getting 2 IP addresses, one static that we set above, and one dynamic via DHCP, turns out the /etc/systemd/network forced the machine to get DHCP, so i went in and commented all the lines inside that file !

Step 3-3: Start the machine and change credentials

Now, after starting the machine, you will need to login to it, to start the virtual machine and do that, issue the command

lxc-start -n vm142 -d
lxc-attach -n vm142

Now, you can use the passwd command to change the container’s password, and you would probably want to install “apt-get install ssh openssh-server”, this way you can login to it with putty or any other SSH client

4- Unprivileged LXC containers (Both)

Whatever in this section applies to unprivileged containers, whether root user or any other user

Step 4-1: Enable Unprivileged User Namespaces

it is enabled by default, To make sure that it is, run the command below, if it returns “kernel.unprivileged_userns_clone = 1” you are good to go.

sysctl kernel.unprivileged_userns_clone

if for any reason it is not enabled (0), you can enable it by adding it to /etc/sysctl.d…. by editing the file “/etc/sysctl.d/00-local-userns.conf” and adding the following line, if the file does not exist, create it


Once done, run the command

service procps restart

5- Unprivileged container under new user

Step 5-1: Create the user

You can call the user whatever you want, I chose to call the user lxcadmin, this is an arbitrary choice, To create a user we issue the following command.

adduser lxcadmin

The output of the adduser command should be something like

Adding user `lxcadmin' ...
Adding new group `lxcadmin' (1001) ...
Adding new user `lxcadmin' (1001) with group `lxcadmin (1001)' ...
Creating home directory `/home/lxcadmin' ...
Copying files from `/etc/skel' ...
Adding new user `lxcadmin' to supplemental / extra groups `users' ...
Adding user `lxcadmin' to group `users' ...

So here, our user gets the ID 1001 (Since i already have a user with the ID 1000 and the root user with the ID 0. Now if we inspect the 2 files /etc/subuid (The subordinate uid file) and /etc/subgid, we will find the following content in both (Identical contents in files).


What the above means is that user lxcadmin has a range of UIDs starting with 165536 and has 65536 extra UIDs total, so the last UID that lxcadmin can use is 165536 + 65536 – 1 = 321071, and the next user we add will start at 321072.

So to recap this user has a subordinate ID range from 165537 TO 321071, notice i added one to the starting number since the first number is not a subordinate ID, but rather the user’s default ID.

Step 5-2: Network adapter quota

New users generally do not have the ability to add a container to a bridge, for that you will need to give the user a network device quota, this quota is defined in the file /etc/lxc/lxc-usernet, the initial quota for unprivileged users is zero, so edit the file and add the following lines, depending on what adapters you would like to allow lxcadmin to connect containers to, the format is user type bridge quota

lxcadmin veth lxcbr0 10
lxcadmin veth br0 10

Notice that you can replace the user with a group name, but that is a subject of a different post…

Now you will need to copy the file /etc/lxc/default.conf to the user’s home directory, in my case under /home/lxcadmin/.config/lxc/default.conf, if the config directory does not exist, create it, now edit this file you just created and depending on the user you are using (I am using the second user, hence the numbers, yours will differ unless your user is the second one added, copy the values from /etc/subuid)…

    lxc.idmap = u 0 165536 65536
    lxc.idmap = g 0 165536 65536

Now, we are closer than ever to making it run, we need to create our first container, unlike the privileged “lxc-create mycontainer” this one is slightly more complicated (The solution is below to make things unprivileged and secure again)

systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-create -t download -n my-container
lxc-create -t download -n myunprivcontainer -- -d debian -r bookworm -a amd64

Don’t expect this to work yet…. the following contgainer config file was automatically created

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: -d debian -r bookworm -a amd64
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64

# Container specific configuration
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
lxc.idmap = u 0 165536 65536
lxc.idmap = g 0 165536 65536
lxc.rootfs.path = dir:/home/lxcadmin/.local/share/lxc/myunprivcontainer/rootfs = myunprivcontainer

# Network configuration = veth = br0 = up

libpam-cgfs is already installed (It was a dependancy in the apt-get install above), libpam-cgfs is a Pluggable Authentication Module (PAM) to provide logged-in users with a set of cgroups which they can administer. This allows for instance unprivileged containers, and session management using cgroup process tracking.

Configure AppArmor

App Armor is enabled on Debian 10 (buster) and after by default, AppArmor is recommended as it adds a layer of security which may prove vital for a system running your virtual machines.

to check whether it is enabled on your system or not, you can run the following command

cat /sys/module/apparmor/parameters/enabled

If the above returns the letter Y, AppArmor is enabled, and you need to set it up to allow for our unprivileged setup

6- Unprivileged container under root subordinates

This is the most interesting setups, It is a no compromise setup where you can have a container run with all the features you see in privileged containers, while still maintaining the security provided by the unprivileged setup above (More or less)

Step 6-1: root Subordinates:

the first step is to allocate a uid and gid range to root user in /etc/subuid and /etc/subgid. This is because the root user, unlike users added with adduser, does not have subordinate IDs by default, so in short, figure out what the next range of IDs is, and assign them to root by adding a line similar to the following at the top of the list in those 2 files, In my case, lxcadmin has the last range, 165536:65536 means the next id is (165536 + 65536 = 231072), And i would like a million subordinate IDs so i can hand every machine a different set of IDs which should increase security even farther.


adduser will recognize the new range when you use it next time, and start from there

And reflect that range in /etc/lxc/default.conf using lxc.idmap entries similar to those above.

root does not need network devices quotas and uses the global configuration file, so those steps from the above are not needed.

Any container you create as root from that point on will be running unprivileged, able to auto-start, and share filesystems !

Tunneling Firefox traffic through SSH – Putty

I will here assume you already have a remote Linux machine that you can SSH into with putty, the instructions are simple from this point on

Putty Setup

1- Basic putty settings, assuming you have already downloaded putty from, now open putty, enter the IP of the server you wish to tunnel through, and save it with a name, the steps are…
– Open putty,
– enter the IP of your remote machine
– give it a name of your choice
– save (You don’t need to save now, you will save again in a bit, but you can do it anyway)

2- Go to Connection and expand it, then expand SSH, then select Tunnels, this will show a dialogue such as the below, fill in the data as follows

  • A Source port between 1025-65536 (of your choice), i chose 8081 but you can chose any other in that range
  • Check Dynamic and Auto, the click Add

3- From the menu on the left, go back to Session, and click the save button again (So that the new tunnel settings are saved for next time)

4- You are almost done, Now double click the saved session name or select it and hit open, the remote machine should now prompt you to enter a username and a password, once you enter those, you have a tunnel ready on your localhost ( on port 8081, next we will setup Firefox to use that tunnel

Firefox setup

1- Go to firefox settings (Click the accordion menu to the right, and chose settings), once open, scroll down under general, until you find the Network Settings section, click the settings button in that section

Clicking settings above will show the following popup dialogue, setup your system as follows

  • Manual Proxy Configuration
  • SOCKS Host enter and in the port area of that the port we chose in putty (In my case, 8081)
  • Optional – Add the IP address ranges of the IPs that you do not want to have tunneled through the remote machine
  • For more privacy, and sometimes functionality (When access is blocked from abroad), make sure you tunnel your DNS queries as well (See checkbox below)

Now, to verify that you are conencted to the remote machine, google the following

what is my ip

and google should tell you what your IP address is, at this stage, it should be the same as the remote machine’s IP (Not yours)

Mounting unclean NTFS windows drive in Linux

Whenever i get the following message

mount /dev/sdd1 /hds/sgt2tb
The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
Could not mount read-write, trying read-only

The command

ntfsfix /dev/sdd1

resolves the issue, and produces the following message

Mounting volume... The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Attempting to correct errors...
Processing $MFT and $MFTMirr...
Reading $MFT... OK
Reading $MFTMirr... OK
Comparing $MFTMirr to $MFT... OK
Processing of $MFT and $MFTMirr completed successfully.
Setting required flags on partition... OK
Going to empty the journal ($LogFile)... OK
Checking the alternate boot sector... OK
NTFS volume version is 3.1.
NTFS partition /dev/sdd1 was processed successfully

The same mount command you see here will now work flawlessly

mount /dev/sdd1 /hds/sgt2tb

I am still unsure what process from the mentioned above is responsible, as this oftentimes pops up on drives that were never system drives, so there is no hibernation file problem

Mounting a remote Linux file system as a Windows drive

You can do this in many ways, the most popular of which is SAMBA, but this is not the software we are using, here we are using SSHFS

The software this post is about is SSHFS, if you are reading this, you probably know what SSH is (Secure shell), and FS stands for File System

Ironically, you will only need to have SFTP and not SSH with shell access, so here is the first surprise, Now, to continue with this tutorial, you might want to visit the page I have posted here to create that user and give him/her access to the directory to be mounted, don’t worry, there is a link back here at the bottom of that page !

So, now that you have created that user account on the remote system, let’s get down to business

You will need 2 peices of software, or 3 if you would like to use private/public key authentication

For the following software, look on their websites for the latest installers for your version of Windows (Usually you are looking for the msi of the 64bit version of windows)

1- WinFsp, short for Windows File System Proxy, What this basically does is enabled the developer of SSHFS-Win to make it look like a windows drive, not some separate SFTP application where you have to move the files manually, when you present it as a drive, you can modify files directly on it, which is the main advantage, and it will do the work in the background, it is a driver that presents itself on/to windows as a disk, while cheating the disk contents from another application, the github page for it is at, or to save you time, Just go directly to the download page here , When presented with optional components, if you are not a developer, you will only ever need the Core package, which is the installer’s default

Once WinFsp is installed, we are done with the part that allows us to display file systems that are not really filesystems, the next step is to have something feed that with data from an actual filesystem somewhere else ! via SFTP, and that software would be

2- SSHFS-Win, which is the system that sits in the middle, between the SFTP server, and WinFsp which is an illusion of a hard drive on your windows machine ! it’s home on github is at, To get the latest from this one, go here and look for the one that says latest (Not pre-release), download and install it

There is no software to install on the remote side, as most Linux systems already have the functionality ! and you have already setup a user in the previous post that I pointed you to a minute ago, So let us mount !

Now, you can (But don’t do it just yet) open file explorer in Windows, right click “This PC”, and click on Map Network Drive, A dialogue appears, enter your connection string, which should be something like


You should then be prompted with a password dialogue box, you enter the SFTP password, and you should now be all set, but why are we not doing this right now ? we are not doing this because when you create files in that drive, they will remotely have rwx permissions for owner, and no permissions for group or others, wo work around this, you need to pass the following arguments to the mount



which is only available via command line and does not survive reboots, a better alternative is to use sshfs-win-manager, which seamlessly mounts those remote file systems using SFTP , the long and short of it is that it just works

Another program that has a different set of permission issues (I can write files, but can’t write to them again even though i own the files on the remote system and the permissions should allow) is SiriKali (, you should be able to find the line to download for your platform here (

SiriKali also allows you to use other types of authentication which are beyond the scope of this post

So in SiriKali, you need to fill the above information, luckily that information is loaded by default.

Remember to select the checkboxes you need,

Static IP on Hyper-v (Debian Guests)

One problem i face when developing using Hyper-v is that I need static addresses, and the default switch keeps changing the ip range

The simplest solution to this is to create a new switch of type internal ! this only connects the virtual machines to each other (Static IP etc…), and can not access the internet

Right after creating an INTERNAL switch in the switch manager, you go to “Manage network adapter settings” on the host computer, and assign an IP such as to the adapter and a subnet of, no gateway, and nothing but those IPs.

Once that is done, you add a second adapter to all the virtual machines, and in the /etc/network/interfaces file, you leave eth0 the way it was (For internet) and add a metric 10 under the last line for eth0, then add the following stanza for the new adapter (Assuming eth1), eth 1 has a higher cost in it’s metric, so unless the remote is on the eth1 subnet, it will go through the eth0

auto eth1
iface eth1 inet static
 metric 100

And you are done, those virtual machines can address each other with their 10.10.20.x addresses, and access the outside world via the other network interface.

Video Editing for Debian 11

On linux, when i needed to edit a video I usually go to Kdenlive, I used to think it is THE linux alternative to Adobe Premier Pro, until yesterday, when a friend recommended I try out OpenShot

I am a Gnome user, Kdenlive was designed for KDE, I have always run it in gnome (Unity) and it worked fine, but today, I am feeling adventurous and up for trying OpenShot, openshot too is pyQt which is a bummer, but hey, this is not why we doing this

KDE is based on Qt and Gnome is based on GTK. both applications here are meant for KDE, and I don’t really want to install KDE on my machine

This post here is where I will leave my impressions so hang tight.

redis for Laravel on Debian

If you are like me, running only your own projects on a server, you might want to skip authentication

the changes I generally make to the file /etc/redis/redis.conf are

1- At the very beginning, limit the RAM redis can use with the line

maxmemory 2gb

2- Change the supervisor to systemd by modifying the line

supervised no


supervised systemd

Now, to test the new config, from the command line, run the following commands

systemctl restart redis-server
config get maxmemory

You have just configured redis and tested your new settings.