{"id":1164,"date":"2023-08-02T21:05:26","date_gmt":"2023-08-03T05:05:26","guid":{"rendered":"https:\/\/angrysysadmins.tech\/?p=1164"},"modified":"2023-08-02T21:05:26","modified_gmt":"2023-08-03T05:05:26","slug":"vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine","status":"publish","type":"post","link":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/","title":{"rendered":"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine"},"content":{"rendered":"

What is ReBAR and Why Do I Want It<\/h2>\n

Resizable BAR is a PCIe setting that allows the device (in this case GPU) to negotiate Base Address Register size (in this case access the whole frame buffer at once rather than in 256MB chunks). Some games make better use of this than others, and older GPUs do not support it. For games and hardware that make good use of ReBAR, the benefits are very nice to have.<\/p>\n

Pre Requisites<\/h2>\n

You need ALL of these for this to work<\/p>\n

    \n
  1. Your linux machine is booted using UEFI.<\/li>\n
  2. Your GPU supports ReBar.<\/li>\n
  3. Your Windows VM is installed and booted using UEFI.<\/li>\n
  4. You are running linux Kernel 6.1 or later.<\/li>\n
  5. You are running QEMU version 7.0.3 or later<\/li>\n
  6. A CPU with 37 bits or more if you have nested pages enabled\n
      \n
    1. Check with egrep -w 'npt|ept' \/proc\/cpuinfo<\/code> (npt is what AMD calls it, EPT is what Intel calls it)<\/li>\n
    2. Check with the command grep 'bits physical' \/proc\/cpuinfo<\/code> If you see numbers greater than or equal to 37 you are in the clear.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

      Host<\/h2>\n

      Set the ReBAR size in command line. Find your GPU with lspci. Mine is listed as:<\/p>\n

      45:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD\/ATI] Navi 21 [Radeon RX 6800\/6800 XT \/ 6900 XT] (rev c0)<\/pre>\n
      Next we need to get the bus address. Use the beginning part of that return, in my case 45:00.0, and run:<\/p>\n
      lspci -n | grep 45:00.0<\/pre>\n
      With that output we now have all of the identifiers we need. In my case the returned result is:<\/p>\n
      45:00.0 0300: 1002:73bf (rev c0)<\/pre>\n
      Now with these Identifiers, we can find out what our current settings are. Run the following command replacing $GPUAddress with your address to see what the ReBar is set to.<\/p>\n
      lspci -vvvs $GPUAddress | grep BAR<\/pre>\n
      My output was<\/p>\n
      #lspci -vvvs 00:45:00.0 | grep BAR 130 \u21b5\r\nCapabilities: [200 v1] Physical Resizable BAR\r\nBAR 0: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB\r\nBAR 2: current size: 2MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB<\/pre>\n
      We can see that my AMD RX 6900 XT ReBar’s are 16GB for BAR0 (the main one) which is good as my video memory is 16GB on my RX 6900 XT. Bar 2 is set to 2MB, its smallest value. I’d like to change this to something higher. Window’s maximum for BAR2 is 8MB. NVIDIA cards have BAR0 set to a single value, and BAR1 is the Main BAR you want to edit. If BAR 2 on an AMD GPU size is greater than 8MB, Windows will not load the device, erroring out with Error Code 43.<\/strong><\/p>\n
       <\/p>\n
      Now this next part is specific to how you use your gaming virtual machine. In my case, I detach my my GPU from my host machine and apply the vfio-pci driver to it. It is detailed in the article on here titled: VFIO: Tuning your Windows gaming VM for optimal performance<\/a>. I have a script that runs at the start of my virtual machine to detach the GPU from the amdgpu driver, apply the ReBar settings, and make it use the vfio-pci driver for safe pass through.<\/p>\n
      Here is a snippet from that script:<\/p>\n
      echo \"unbind 6900xt gpu from amdgpu (1002:73bf)\"\r\necho 0000:45:00.0 > \/sys\/bus\/pci\/drivers\/amdgpu\/unbind\r\nsleep 1\r\n\r\necho \"Setting rebar 0 size to 16GB\" \r\necho 14 > \/sys\/bus\/pci\/devices\/0000:45:00.0\/resource0_resize\r\nsleep 1\r\n\r\necho \"Setting the rebar 2 size to 8MB\"\r\n#Driver will error code 43 if above 8MB on BAR2 \r\necho 3 > \/sys\/bus\/pci\/devices\/0000:45:00.0\/resource2_resize\r\n\r\nsleep 2\r\necho 1002 73bf > \/sys\/bus\/pci\/drivers\/vfio-pci\/new_id || echo -n \"0000:45:00.0\" > \/sys\/bus\/pci\/drivers\/vfio-pci\/bind\r\necho done\r\nsleep 1<\/pre>\n
      I am echoing the bit value of the ReBAR size I want. Values go as follows:<\/p>\n
      Bit Sizes\r\n1 = 2MB\r\n2 = 4MB\r\n3 = 8MB\r\n4 = 16MB\r\n5 = 32MB\r\n6 = 64MB\r\n7 = 128MB\r\n8 = 256MB\r\n9 = 512MB\r\n10 = 1GB\r\n11 = 2GB\r\n12 = 4GB\r\n13 = 8GB\r\n14 = 16GB\r\n15 = 32GB<\/pre>\n
      When setting the bar value, do not exceed the vram of your machine for BAR0 on amd or BAR1 on nvidia. There are ways to do this in kernel parameters and modprobe commands, but this is the method I use.<\/p>\n
      Virtual Machine<\/h2>\n
      Here are the changes you need to make to your libvirt configuration<\/p>\n
        \n
      1. Change the LibVirt Domain from <domain type=\"kvm\/><\/code> to <domain xmlns:qemu=\"http:\/\/libvirt.org\/schemas\/domain\/qemu\/1.0\" type=\"kvm\"><\/code><\/li>\n
      2. Append this to the bottom of your config:\n
        <\/devices>\r\n<qemu:commandline>\r\n<qemu:arg value=\"-fw_cfg\"\/>\r\n<qemu:arg value=\"opt\/ovmf\/X-PciMmio64Mb,string=65536\"\/>\r\n<\/qemu:commandline>\r\n<\/domain><\/pre>\n<\/li>\n<\/ol>\n
        With those changes you can now boot your Virtual Machine. Download a program called GPU-Z to check if ReBar is working and enabled. GPU-Z will say Resizable Bar: Enabled<\/p>\n","protected":false},"excerpt":{"rendered":"
        What is ReBAR and Why Do I Want It Resizable BAR is a PCIe setting that allows the device (in this case GPU) to negotiate Base Address Register size (in this case access the whole frame buffer at once rather than in 256MB chunks). Some games make better use of this than others, and older 
        Read More »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,22],"tags":[],"coauthors":[39],"class_list":["post-1164","post","type-post","status-publish","format-standard","hentry","category-linux","category-virtualization"],"yoast_head":"\nVFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine - Angry Sysadmins<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine - Angry Sysadmins\" \/>\n<meta property=\"og:description\" content=\"What is ReBAR and Why Do I Want It Resizable BAR is a PCIe setting that allows the device (in this case GPU) to negotiate Base Address Register size (in this case access the whole frame buffer at once rather than in 256MB chunks). Some games make better use of this than others, and older Read More »\" \/>\n<meta property=\"og:url\" content=\"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/\" \/>\n<meta property=\"og:site_name\" content=\"Angry Sysadmins\" \/>\n<meta property=\"article:published_time\" content=\"2023-08-03T05:05:26+00:00\" \/>\n<meta name=\"author\" content=\"Ryan Parker\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ryan Parker\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/\"},\"author\":{\"name\":\"Ryan Parker\",\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/#\\\/schema\\\/person\\\/651321cd35645fb6a4d8a75b7bc7c199\"},\"headline\":\"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine\",\"datePublished\":\"2023-08-03T05:05:26+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/\"},\"wordCount\":570,\"commentCount\":8,\"articleSection\":[\"Linux\",\"Virtualization\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/\",\"url\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/\",\"name\":\"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine - Angry Sysadmins\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/#website\"},\"datePublished\":\"2023-08-03T05:05:26+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/#\\\/schema\\\/person\\\/651321cd35645fb6a4d8a75b7bc7c199\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/2023\\\/08\\\/grassyloki\\\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/angrysysadmins.tech\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/#website\",\"url\":\"https:\\\/\\\/angrysysadmins.tech\\\/\",\"name\":\"Angry Sysadmins\",\"description\":\"A site full of angry sysadmins here to vent and help\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/angrysysadmins.tech\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/angrysysadmins.tech\\\/#\\\/schema\\\/person\\\/651321cd35645fb6a4d8a75b7bc7c199\",\"name\":\"Ryan Parker\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/fc12b1a02765c8017062ee6f41eb34a7b14575bcd8acd7da40e176fe8f12b10f?s=96&d=mm&r=g664d0e05248e51cb1d71b3f66c6f929d\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/fc12b1a02765c8017062ee6f41eb34a7b14575bcd8acd7da40e176fe8f12b10f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/fc12b1a02765c8017062ee6f41eb34a7b14575bcd8acd7da40e176fe8f12b10f?s=96&d=mm&r=g\",\"caption\":\"Ryan Parker\"},\"description\":\"Professionally im a Infrastructure Security Specialist. I current maintain a homelab with about 3TB of RAM, 240+ TB of storage, tons of CPU cores, and 100gbit networking backbone in the garage running up my electricity bill.\",\"url\":\"https:\\\/\\\/angrysysadmins.tech\\\/index.php\\\/author\\\/grassyloki\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine - Angry Sysadmins","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/","og_locale":"en_US","og_type":"article","og_title":"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine - Angry Sysadmins","og_description":"What is ReBAR and Why Do I Want It Resizable BAR is a PCIe setting that allows the device (in this case GPU) to negotiate Base Address Register size (in this case access the whole frame buffer at once rather than in 256MB chunks). Some games make better use of this than others, and older Read More »","og_url":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/","og_site_name":"Angry Sysadmins","article_published_time":"2023-08-03T05:05:26+00:00","author":"Ryan Parker","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ryan Parker","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/#article","isPartOf":{"@id":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/"},"author":{"name":"Ryan Parker","@id":"https:\/\/angrysysadmins.tech\/#\/schema\/person\/651321cd35645fb6a4d8a75b7bc7c199"},"headline":"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine","datePublished":"2023-08-03T05:05:26+00:00","mainEntityOfPage":{"@id":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/"},"wordCount":570,"commentCount":8,"articleSection":["Linux","Virtualization"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/","url":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/","name":"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine - Angry Sysadmins","isPartOf":{"@id":"https:\/\/angrysysadmins.tech\/#website"},"datePublished":"2023-08-03T05:05:26+00:00","author":{"@id":"https:\/\/angrysysadmins.tech\/#\/schema\/person\/651321cd35645fb6a4d8a75b7bc7c199"},"breadcrumb":{"@id":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/angrysysadmins.tech\/index.php\/2023\/08\/grassyloki\/vfio-how-to-enable-resizeable-bar-rebar-in-your-vfio-virtual-machine\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/angrysysadmins.tech\/"},{"@type":"ListItem","position":2,"name":"VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine"}]},{"@type":"WebSite","@id":"https:\/\/angrysysadmins.tech\/#website","url":"https:\/\/angrysysadmins.tech\/","name":"Angry Sysadmins","description":"A site full of angry sysadmins here to vent and help","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/angrysysadmins.tech\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/angrysysadmins.tech\/#\/schema\/person\/651321cd35645fb6a4d8a75b7bc7c199","name":"Ryan Parker","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/fc12b1a02765c8017062ee6f41eb34a7b14575bcd8acd7da40e176fe8f12b10f?s=96&d=mm&r=g664d0e05248e51cb1d71b3f66c6f929d","url":"https:\/\/secure.gravatar.com\/avatar\/fc12b1a02765c8017062ee6f41eb34a7b14575bcd8acd7da40e176fe8f12b10f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/fc12b1a02765c8017062ee6f41eb34a7b14575bcd8acd7da40e176fe8f12b10f?s=96&d=mm&r=g","caption":"Ryan Parker"},"description":"Professionally im a Infrastructure Security Specialist. I current maintain a homelab with about 3TB of RAM, 240+ TB of storage, tons of CPU cores, and 100gbit networking backbone in the garage running up my electricity bill.","url":"https:\/\/angrysysadmins.tech\/index.php\/author\/grassyloki\/"}]}},"_links":{"self":[{"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/posts\/1164","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/comments?post=1164"}],"version-history":[{"count":7,"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/posts\/1164\/revisions"}],"predecessor-version":[{"id":1200,"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/posts\/1164\/revisions\/1200"}],"wp:attachment":[{"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/media?parent=1164"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/categories?post=1164"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/tags?post=1164"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/angrysysadmins.tech\/index.php\/wp-json\/wp\/v2\/coauthors?post=1164"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}