Asus Z10PE-D16 WS C612 - ReBAR requires pci=realloc in linux and does not really work in Win10

[LegionnaireENR34]

System

• Motherboard: Asus Z10PE-D16 WS - this is dual 2011-3 socket board on C612 PCH
• Two Xeon e5-2600v3 (haswell-e) installed
• 128GB RAM
• BIOS Version: 4101
• GPU: Intel Arc A770 16GB VRAM attached to NUMA node 2 with x16 link
• CSM is turned off in BIOS
• 4G decoding is enabled in BIOS
• UEFIPatch is applied
• DSDT Patch not applied as I assumed it is not needed
• I have read Common issues (and fixes)

Description

Hey, I'd like to ask for an advice and a feedback. I run Z10PE-D16 WS - it is a dual socket board with 16 memory slots and c612 chipset. BIOS used for modification is v.4101, latest available at asus.com
The intention was to make ReBAR work with ARC A770

Current state:

1. I've installed UEFI .ffs module as per instructions
2. UEFI patch applied. The one applied was about whitelisting on x99/c612 motherboads, as reported by hex string
3. Above 4G decoding enabled
4. CSM disabled

First step
of enabling resizable BAR was performed in Linux Mint with Kernel v.6.5

• If rebarstate variable is set to 0, dmesg shows 256M register size, i915 kernel driver loads successfully
• If rebarstate variable is set at any value ranging from 14 to 32, lspci reports that intended BAR size is reported to be 16G in lspci, but i915 kernel driver does not load, dmesg shows conflicts in addresses, stating that there is not enough space to allocate BAR of required size.
• If I add pci=realloc to kernel boot parameters, then any value from 14 to 32 in rebarstate works, assigned BAR size is 16G, kernel mode i915 driver is loaded successuly, but I see a bunch of address space reallocations in dmesg.
  So to that end adding ReBAR appears to be working provided that pci=realloc is configured during linux boot. However I have an impression that reallocation should not be required.

Second step
Now, to confirm that ReBAR was really working I tried to install a clean Windows 10. At this time Rebarstate is set at 32.

• Installation went smoothly, OS booted fine.
• Then I've installed Intel's GPU driver. Immediately after installation and before reboot it complained about insufficient resources for the driver to start in the device manager. After reboot Win10 hangs during boot process.
• When rebarstate is set to 8, win10 boots fine as it should with the default 256MB BAR
• When rebarstate is set to 9, it's the same
• When rebarstate is set to 10, motherboard passes post 1 time out of 3, diagnostic codes are pretty messed up
• Rebarstate set to 11 would send the board to cyclic reboot until GPU is unplugged, it wouldn't post
• When rebarstate is set to 12, MB would post, but win10 would BSOD
• With rebar of 13 and above win10 just hangs during boot, no BSOD, no recovery, just locks up requiring hard reset

As a side note, installing win10 with all associated drivers and playing with rebarstate messed post codes. Before win10 post codes as seen on LED display on the MB matched the ones shown on my monitor. After win10 and drivers only onboard display shows correct codes, monitor just shows A9 "Start of setup" code till the end of the post.

So is there something wrong with either resource allocation, or with the way .ffs module behaves on dual-CPU/c612 platforms? I guess this result cannot be considered a complete failure, and yet I would not regard it as a success either. Have I done anything wrong? Can you please share any thoughts or suggestions as to what should be done next? I can provide dmesg or lspci output if needed.

[xCuri0]

Try changing BIOS settings related to MMIOH, I know X99/C612 have a few options relating to it under IntelRCSetup.

these boards should work fine with rebar when configured properly, they're made to be used with tesla/a gpus that require large bar after all

[LegionnaireENR34]

Thanks for the advice, will try that. Found two options under IntelRCSetup -> Common RefCode Configuration. They are hidden by default, will need to expose them in AMIBCP in order to play with:

• MMIOH Base - seems to default at 56T, options: 56T, 40T, 24T, 16T, 12T, 4T, 2T, 1T
• MMIO High size - seems to default at 256G, options: 128G, 256G, 512G, 1024G
  Both default values seem pretty high already, should I reduce them? Or try to set MMIOH size to 512G and above and leave 56T Base value alone?

There are other options in BIOS that are PCIe-port specific, about single 64-bit bar or two split 32-bit bars, I tried to play with them briefly, but as far as I can tell those don't affect my problem.

Another question that interests me: does it matter which CPU the GPU is connected to? Do they both share common address space? Do QPI settings and limitations play any role in ReBAR at all?

[LegionnaireENR34]

Hey @xCuri0, so I've done some additional testing.
I have exposed MMIO-related options that seemed to be relevant in BIOS, there were 3 of them:

• MMCFG base - set to 2G by default
• MMIOH base- set to 56T by default
• MMIOH size - there are 4 sizes: 128G, 256G, 512G, 1024G.

I played with all 4 of the "MMIOH size", but none worked without pci=realloc.
Here's dmesg report of the result. Device 85:00 is Intel Arc A770 16G:

If i'm reading this correctly, it appears to be trying to allocate memory starting from address 0x0 and towards higher addresses, not the other way around. In trying to do so it displaces some other devices' memory ranges.

Any ideas as to what I should try next? Any other options to look for in bios?
Why is it trying to allocate memory from low addresses? Is there a way to change this behavior? Is it defined by device driver, OS or UEFI itself?
Can this issue be relevant to the system being two-CPU and some limitations of how memory is accessed via QPI?
Should I try different brand GPU?

[LegionnaireENR34]

@xCuri0
Some new development here. To ensure that my issues were not hardware-specific, I've managed to obtain AMD RX6900 for testing.
Can report that is was at least some success.

• On Windows 10 with RX6900 and this UEFI module ReBAR works successfully with any rebarstate variable above 12. Here are the screenshots of it with rebarstate at 32:

• On Linux Mint kernel v.6.5 with the same rebarstate amdgpu driver fails to start without pci=realloc kernel parameter - it gets stuck in permanent "amdgpu trn=2 ack should not assert! wait again!" message in dmesg. With pci=realloc it appears to load successfully.

So As far as ReBarUEFI with Z10PE-D16 WS, we can declare partial success, but not full success.
Whether the new info justifies any updates to this UEFI module or not - not for me to decide. I want to thank you for you work on it.
As of now, I think we can close this thread.

[OdinVex]

UEFI patch applied. The one applied was about whitelisting on x99/c612 motherboads, as reported by hex string

I have a board that's C612 and although UEFIPatch initially wouldn't patch anything I'm curious which patch you applied.

Edit: I happen to have a dual-CPU/C612 and I'm trying for two different GPUs, an Nvidia Quadro P4000 (8GB) and an Intel Arc 310 (4GB). I haven't been able to get the bar resized but I'm using Linux and checking via nvidia-smi -q -x and dmesg. Also, isn't our E5-26XXv4 CPUs and this chipset Broadwell-EP (is that different from Broadwell?) I come from AMD-land, so I'm unaware of a lot of Intel's chipsets. I wonder if the boards try to put MMIO space above 64GB.

[LegionnaireENR34]

I have a board that's C612 and although UEFIPatch initially wouldn't patch anything I'm curious which patch you applied.

As you can tell, it was quite some time ago, so my recollection is a bit vague about specifics, but I'm pretty certain that it was this one that was found and updated:
# AMI APTIO V NvramSmiDxe (Socket 2011-v3 MB: C612, X99) NVRAM whitelist unlock 54B070F3-9EB8-47CC-ADAF-39029C853CBB 10 P:0F84B300000041F6:90E9B300000041F6

Edit: I happen to have a dual-CPU/C612 and I'm trying for two different GPUs, an Nvidia Quadro P4000 (8GB) and an Intel Arc 310 (4GB). I haven't been able to get the bar resized but I'm using Linux and checking via nvidia-smi -q -x and dmesg.

P4000 is pascal generation, isn't it? If it is, I'd say it is not the correct GPU to check for ReBAR, as I'm quite certain that pascal and its drivers are not aware of resizable BAR. Arc A380 should be the GPU to check is ReBAR works - be mindful of kernel memory allocations though, some extra parameters might be required as I referred above.

Also, isn't our E5-26XXv4 CPUs and this chipset Broadwell-EP (is that different from Broadwell?) I come from AMD-land, so I'm unaware of a lot of Intel's chipsets.

Sorry, I don't understand the question. E5-26xx v3/v4 and C61x are indeed Haswell-EP and Broadwell-EP, in the sense that they use hsw/brw core, but otherwise they differ. It goes as follows:
Haswell/Broadwell for desktop:

• socket 1150,
• Z/Q/B/H 80/90-series chipset, or C22x for Xeon E3 v3,
• 2-channel DDR3 memory controller,
• 16 PCIe lanes.

Haswell/Broadwell-E for HEDT:

• socket 2011-3,
• x99 chipset,
• 4-channel DDR4 memory controller,
• 28/40 PCIe lanes.

Haswell/Broadwell-EP for servers:

• socket 2011-3,
• c61x chipset,
• 4-channel DDR4 memory controller,
• 28/40 PCIe lanes,
• can have QPI for multi-CPU config, so 1-, 2- and 4-socket motherboards available.

I wonder if the boards try to put MMIO space above 64GB.

As far as I'm aware, there's nothing wrong with putting MMIO region above 64GB (which is not a lot of space actually) provided that Above 4G decoding is enabled. MMIOH base in my case was way above 64GB, it was at 56T

[OdinVex]

As you can tell, it was quite some time ago, so my recollection is a bit vague about specifics, but I'm pretty certain that it was this one that was found and updated: # AMI APTIO V NvramSmiDxe (Socket 2011-v3 MB: C612, X99) NVRAM whitelist unlock 54B070F3-9EB8-47CC-ADAF-39029C853CBB 10 P:0F84B300000041F6:90E9B300000041F6

UEFITool never sees those bytes to patch. I'll try to find some like-patternness in that DXE, maybe my BIOS needs a slightly-tweaked one.

P4000 is pascal generation, isn't it? If it is, I'd say it is not the correct GPU to check for ReBAR, as I'm quite certain that pascal and its drivers are not aware of resizable BAR. Arc A380 should be the GPU to check is ReBAR works - be mindful of kernel memory allocations though, some extra parameters might be required as I referred above.

I wouldn't have a clue, I left Nvidia around the GTX480 series. It has an upgraded UEFI-capable BIOS and Nvidia drivers report it has a bar size but it's just capped to 256MB. The Intel Arc 310 is UEFI-enabled from factory and can use rebar as well, according to Intel anyway. It's pretty much required for any performance out of it. Granted the 380's faster and better but I wanted one that could survive on PCIe power alone. I might upgrade but I'd rather try if I can. It can handle enough transcoding streams simultaneous for what I wanted.

Sorry, I don't understand the question. E5-26xx v3/v4 and C61x are indeed Haswell-EP and Broadwell-EP, in the sense that they use hsw/brw core, but otherwise they differ.

Haswell/Broadwell-EP server then, it's a humble Supermicro SYS-1028-TR4T+.

As far as I'm aware, there's nothing wrong with putting MMIO region above 64GB (which is not a lot of space actually) provided that Above 4G decoding is enabled. MMIOH base in my case was way above 64GB, it was at 56T

The default it was giving me was also 56T, I can't recall more about the MMIO configuration. Above 4G Decoding is enabled.

I wonder if that NVRAM patch not being found is an issue. The USB3 patches, none of them would be relevant? I recall finding a very similar byte-sequence that only differed in the PCI ID of the chipset being blocklisted. I never tried adjusting and patching that, though. I haven't used USB at all on the board (except via IPMI's 2.0, etc).

When erasing via IPMI/BMC, should I erase the SMBIOS as well? That's always left checked by default in the upgrade. Thank you very much for you reply, it's been wracking to try to figure this board out.

Edit: Just dumped my AMI APTIO V NvramSmiDxe 54B070F3-9EB8-47CC-ADAF-39029C853CBB, nothing even remotely close to a matching byte-pattern (even after decompression). Is this just so ReBar can read/write to NVRAM? I don't mind hard-coding the value and removing NVRAM-stuff from ReBar if that'd work around it.

Edit: Nvidia's Quadro P4000 8GB does appear to be Pascal. I don't usually touch Nvidia so I'm unaware of the implications. It does support aperture sizes from what I see online but I guess it might just not have the support it should. For now I'll focus on the Intel Arc. I wonder if having both in might affect resizing efforts by ReBar or Linux?