VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine

What is ReBAR and Why Do I Want It

Resizable BAR is a PCIe setting that allows the device (in this case GPU) to negotiate Base Address Register size (in this case access the whole frame buffer at once rather than in 256MB chunks). Some games make better use of this than others, and older GPUs do not support it. For games and hardware that make good use of ReBAR, the benefits are very nice to have.

Pre Requisites

You need ALL of these for this to work

1. Your linux machine is booted using UEFI.
2. Your GPU supports ReBar.
3. Your Windows VM is installed and booted using UEFI.
4. You are running linux Kernel 6.1 or later.
5. You are running QEMU version 7.0.3 or later
6. A CPU with 37 bits or more if you have nested pages enabled
   1. Check with egrep -w 'npt|ept' /proc/cpuinfo (npt is what AMD calls it, EPT is what Intel calls it)
   2. Check with the command grep 'bits physical' /proc/cpuinfo If you see numbers greater than or equal to 37 you are in the clear.

Host

Set the ReBAR size in command line. Find your GPU with lspci. Mine is listed as:

45:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c0)

Next we need to get the bus address. Use the beginning part of that return, in my case 45:00.0, and run:

lspci -n | grep 45:00.0

With that output we now have all of the identifiers we need. In my case the returned result is:

45:00.0 0300: 1002:73bf (rev c0)

Now with these Identifiers, we can find out what our current settings are. Run the following command replacing $GPUAddress with your address to see what the ReBar is set to.

lspci -vvvs $GPUAddress | grep BAR

My output was

#lspci -vvvs 00:45:00.0 | grep BAR 130 ↵
Capabilities: [200 v1] Physical Resizable BAR
BAR 0: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
BAR 2: current size: 2MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB

We can see that my AMD RX 6900 XT ReBar’s are 16GB for BAR0 (the main one) which is good as my video memory is 16GB on my RX 6900 XT. Bar 2 is set to 2MB, its smallest value. I’d like to change this to something higher. Window’s maximum for BAR2 is 8MB. NVIDIA cards have BAR0 set to a single value, and BAR1 is the Main BAR you want to edit. If BAR 2 on an AMD GPU size is greater than 8MB, Windows will not load the device, erroring out with Error Code 43.

Now this next part is specific to how you use your gaming virtual machine. In my case, I detach my my GPU from my host machine and apply the vfio-pci driver to it. It is detailed in the article on here titled: VFIO: Tuning your Windows gaming VM for optimal performance. I have a script that runs at the start of my virtual machine to detach the GPU from the amdgpu driver, apply the ReBar settings, and make it use the vfio-pci driver for safe pass through.

Here is a snippet from that script:

echo "unbind 6900xt gpu from amdgpu (1002:73bf)"
echo 0000:45:00.0 > /sys/bus/pci/drivers/amdgpu/unbind
sleep 1

echo "Setting rebar 0 size to 16GB" 
echo 14 > /sys/bus/pci/devices/0000:45:00.0/resource0_resize
sleep 1

echo "Setting the rebar 2 size to 8MB"
#Driver will error code 43 if above 8MB on BAR2 
echo 3 > /sys/bus/pci/devices/0000:45:00.0/resource2_resize

sleep 2
echo 1002 73bf > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:45:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo done
sleep 1

I am echoing the bit value of the ReBAR size I want. Values go as follows:

Bit Sizes
1 = 2MB
2 = 4MB
3 = 8MB
4 = 16MB
5 = 32MB
6 = 64MB
7 = 128MB
8 = 256MB
9 = 512MB
10 = 1GB
11 = 2GB
12 = 4GB
13 = 8GB
14 = 16GB
15 = 32GB

When setting the bar value, do not exceed the vram of your machine for BAR0 on amd or BAR1 on nvidia. There are ways to do this in kernel parameters and modprobe commands, but this is the method I use.

Virtual Machine

Here are the changes you need to make to your libvirt configuration

1. Change the LibVirt Domain from <domain type="kvm/> to <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
2. Append this to the bottom of your config:

   </devices>
   <qemu:commandline>
   <qemu:arg value="-fw_cfg"/>
   <qemu:arg value="opt/ovmf/X-PciMmio64Mb,string=65536"/>
   </qemu:commandline>
   </domain>

With those changes you can now boot your Virtual Machine. Download a program called GPU-Z to check if ReBar is working and enabled. GPU-Z will say Resizable Bar: Enabled