What is ReBAR and Why Do I Want It
Resizable BAR is a PCIe setting that allows the device (in this case GPU) to negotiate Base Address Register size (in this case access the whole frame buffer at once rather than in 256MB chunks). Some games make better use of this than others, and older GPUs do not support it. For games and hardware that make good use of ReBAR, the benefits are very nice to have.
You need ALL of these for this to work
- Your linux machine is booted using UEFI.
- Your GPU supports ReBar.
- Your Windows VM is installed and booted using UEFI.
- You are running linux Kernel 6.1 or later.
- You are running QEMU version 7.0.3 or later
- A CPU with 37 bits or more if you have nested pages enabled
- Check with
egrep -w 'npt|ept' /proc/cpuinfo(npt is what AMD calls it, EPT is what Intel calls it)
- Check with the command
grep 'bits physical' /proc/cpuinfoIf you see numbers greater than or equal to 37 you are in the clear.
- Check with
Set the ReBAR size in command line. Find your GPU with lspci. Mine is listed as:
45:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c0)
Next we need to get the bus address. Use the beginning part of that return, in my case 45:00.0, and run:
lspci -n | grep 45:00.0
With that output we now have all of the identifiers we need. In my case the returned result is:
45:00.0 0300: 1002:73bf (rev c0)
Now with these Identifiers, we can find out what our current settings are. Run the following command replacing $GPUAddress with your address to see what the ReBar is set to.
lspci -vvvs $GPUAddress | grep BAR
My output was
#lspci -vvvs 00:45:00.0 | grep BAR 130 ↵ Capabilities: [200 v1] Physical Resizable BAR BAR 0: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB BAR 2: current size: 2MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
We can see that my AMD RX 6900 XT ReBar’s are 16GB for BAR0 (the main one) which is good as my video memory is 16GB on my RX 6900 XT. Bar 2 is set to 2MB, its smallest value. I’d like to change this to something higher. Window’s maximum for BAR2 is 8MB. NVIDIA cards have BAR0 set to a single value, and BAR1 is the Main BAR you want to edit. If BAR 2 on an AMD GPU size is greater than 8MB, Windows will not load the device, erroring out with Error Code 43.
Now this next part is specific to how you use your gaming virtual machine. In my case, I detach my my GPU from my host machine and apply the vfio-pci driver to it. It is detailed in the article on here titled: VFIO: Tuning your Windows gaming VM for optimal performance. I have a script that runs at the start of my virtual machine to detach the GPU from the amdgpu driver, apply the ReBar settings, and make it use the vfio-pci driver for safe pass through.
Here is a snippet from that script:
echo "unbind 6900xt gpu from amdgpu (1002:73bf)" echo 0000:45:00.0 > /sys/bus/pci/drivers/amdgpu/unbind sleep 1 echo "Setting rebar 0 size to 16GB" echo 14 > /sys/bus/pci/devices/0000:45:00.0/resource0_resize sleep 1 echo "Setting the rebar 2 size to 8MB" #Driver will error code 43 if above 8MB on BAR2 echo 3 > /sys/bus/pci/devices/0000:45:00.0/resource2_resize sleep 2 echo 1002 73bf > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:45:00.0" > /sys/bus/pci/drivers/vfio-pci/bind echo done sleep 1
I am echoing the bit value of the ReBAR size I want. Values go as follows:
Bit Sizes 1 = 2MB 2 = 4MB 3 = 8MB 4 = 16MB 5 = 32MB 6 = 64MB 7 = 128MB 8 = 256MB 9 = 512MB 10 = 1GB 11 = 2GB 12 = 4GB 13 = 8GB 14 = 16GB 15 = 32GB
When setting the bar value, do not exceed the vram of your machine for BAR0 on amd or BAR1 on nvidia. There are ways to do this in kernel parameters and modprobe commands, but this is the method I use.
Here are the changes you need to make to your libvirt configuration
- Change the LibVirt Domain from
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
- Append this to the bottom of your config:
</devices> <qemu:commandline> <qemu:arg value="-fw_cfg"/> <qemu:arg value="opt/ovmf/X-PciMmio64Mb,string=65536"/> </qemu:commandline> </domain>
With those changes you can now boot your Virtual Machine. Download a program called GPU-Z to check if ReBar is working and enabled. GPU-Z will say Resizable Bar: Enabled