Technology1 min read

Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux

By · Published by Everything Blog

In short

The AMD ROCm platform demands complete 32 GB address mapping per card to facilitate direct device-to-device transfers, but integrating multiple Radeon AI PRO R9700 cards alongside two RTX 3090s triggers initial startup failures because the BIOS requires 64 GB of space for each 32 GB BAR. Delegating allocation to the Linux kernel fails due to a Linux 7.0 bridge window sizing regression that ignores doorbell BAR alignment gaps, sizing the bridge at 128.6 GB instead of the necessary 224 GB. While maintainer Ilpo Järvinen has authored an unmerged 14-patch kernel series to repair the sizing bug, administrators can currently rely on an out-of-tree early boot module to manually configure the layou…

Key points

  • Direct peer-to-peer data transfers on AMD ROCm require that each installed graphics card…: Direct peer-to-peer data transfers on AMD ROCm require that each installed graphics card map its entire 32 GB memory allocation.
  • An unmerged 14-patch kernel series from Ilpo Järvinen successfully fixes the bridge sizin…: An unmerged 14-patch kernel series from Ilpo Järvinen successfully fixes the bridge sizing calculations for multiple Radeon AI PRO R9700 cards.
  • Systems utilizing the Broadcom PEX88096 switch can temporarily bypass the upstream alloca…: Systems utilizing the Broadcom PEX88096 switch can temporarily bypass the upstream allocation failure by loading an early-boot GitHub module.

Sources checked by our analysis pipeline: 1

My PCIe switch is a Broadcom PEX88096 in the board’s one x16 slot, with four AMD Radeon AI PRO R9700s and the two RTX 3090s from my PCIe splitter post behind it, and the build post has the whole box.

The switch gives each card eight lanes and lets the cards copy to each other without going through the CPU. AMD’s ROCm calls that peer-to-peer (P2P), and it only turns it on when each card’s whole 32 GB is mapped. On Linux, that means a 32 GB Resizable BAR on every card. Resizable BAR is the BIOS option that lets a card’s BAR, the range of the CPU’s physical address space that maps onto its memory, be the full 32 GB instead of the default 256 MB.

A few people asked for the module after the build post, so it’s on GitHub as

gpu-rebar-layout. It leans on kernel internals that

could change in any release, so read the README before you load it. A patch series on the mailing

list fixes the sizing bug, but this board will still need…

Technology