I recently bought an Intel Arc Pro B60, and it seems the most popular way of changing the split between the PF and VFs is to allocate memory on the host before enabling the VFs. However, this is an awful hack which I didn’t like, and it also doesn’t allow for variable memory allocations per VF. So I decided to check if there’s a better way, and there is!
If it’s not obvious, this is for cards that support SR-IOV, nothing here will enable SR-IOV on cards that don’t support it.
SysFS (Kernel 7.1 and above)
This is untested as Proxmox is on 7.0 at the time of writing. I will try to update this when I can.
The amount of memory allocated to each VF can be adjusted by writing to the following file:
echo 2147483648 | sudo tee /sys/bus/pci/drivers/xe/0000\:??\:??.0/sriov_admin/vf1/profile/vram_quota
Replace 2147483648 (2 GiB) with the amount of memory you want to allocate to the VF in bytes, 0000\:??\:??.0 with the PCI address of your PF, and vf1 with the VF you want to change.
Alternatively you can set all VFs to the same amount of memory by writing to:
echo 2147483648 | sudo tee /sys/bus/pci/drivers/xe/0000\:??\:??.0/sriov_admin/.bulk_profile/vram_quota
This is documented here. You can also set the scheduling priority too.
DebugFS (Kernel 6.19 and above)
This is a little bit more complicated, and it requires debugfs to be mounted.
Simple way
For an easy shortcut, enable your VFs first, this will allow auto provisioning to do most of the work, and then you can change the memory afterwards:
# Make sure auto provisioning is enabled (it is on boot)
echo 1 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/restore_auto_provisioning
# Enable the VFs
echo 7 | sudo tee /sys/bus/pci/drivers/xe/0000\:??\:??.0/sriov_numvfs
# Set the memory allocation, this will disable auto provisioning but current values are kept
echo 2147483648 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/vf*/tile0/gt0/lmem_quota
Replace 2147483648 (2 GiB) with the amount of memory you want to allocate to the VF in bytes, 0000\:??\:??.0 with the PCI address of your PF, and vf* with the VF you want to change. (Or don’t, and it’ll set all VFs to the same amount of 2147483648, although the wildcards will require a root shell.)
Advanced way
If you want to change the memory allocation without enabling the VFs first, you have to also configure the GGTT and GuC contexts/doorbells.
# Check your available GGTT size, this will be the amount you divide between your VFs
sudo grep ^avail /sys/kernel/debug/dri/0000\:??\:??.0/sriov/pf/tile0/ggtt_available
# Set the amount of GGTT to allocate in bytes (e.g. 512MiB)
echo 536870912 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/vf*/tile0/gt0/ggtt_quota
# Set the amount of VRAM you want to allocate in bytes (e.g. 2GiB)
echo 2147483648 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/vf*/tile0/gt0/lmem_quota
# Set the amount of GuC contexts and doorbells to allocate
echo 8192 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/vf*/tile0/gt0/contexts_quota
echo 32 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/vf*/tile0/gt0/doorbells_quota
echo 8192 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/vf*/tile0/gt1/contexts_quota
echo 32 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/vf*/tile0/gt1/doorbells_quota
# Enable the VFs
echo 7 | sudo tee /sys/bus/pci/drivers/xe/0000\:??\:??.0/sriov_numvfs
# Without auto provisioning enabled, you have to manually set the GuC contexts and doorbells for each VF
Checking Resource Usage
You can check resource provisioning for each VF by reading the following files:
sudo cat /sys/kernel/debug/dri/0000\:??\:??.0/sriov/pf/tile0/gt0/lmem_provisioned
sudo cat /sys/kernel/debug/dri/0000\:??\:??.0/sriov/pf/tile0/gt0/ggtt_provisioned
sudo cat /sys/kernel/debug/dri/0000\:??\:??.0/sriov/pf/tile0/gt*/contexts_provisioned
sudo cat /sys/kernel/debug/dri/0000\:??\:??.0/sriov/pf/tile0/gt*/doorbells_provisioned
You can also check dmesg:
xe 0000:83:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 536870912 (512 MiB) GGTT
xe 0000:83:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 2147483648 (2.00 GiB) LMEM
xe 0000:83:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 8192 GuC context IDs
xe 0000:83:00.0: [drm] PF: Tile0: GT0: VF1 provisioned with 32 GuC doorbell IDs
xe 0000:83:00.0: [drm] PF: Tile0: GT1: VF1 provisioned with 8192 GuC context IDs
xe 0000:83:00.0: [drm] PF: Tile0: GT1: VF1 provisioned with 32 GuC doorbell IDs
Disabling VFs
Disabling the VFs does not automatically reset the resource allocations with auto provisioning disabled, this means you can re-enable the VFs later, and they will still have the same resource allocations. If you want to reset the resource allocations, a quick way is to just re-enable auto provisioning:
# Disable the VFs first
echo 0 | sudo tee /sys/bus/pci/drivers/xe/0000\:??\:??.0/sriov_numvfs
# Re-enable auto provisioning
echo 1 | sudo tee /sys/kernel/debug/dri/0000\:??\:??.0/sriov/restore_auto_provisioning
Setting all the *_quota values to 0 will also work.
Old DebugFS (Kernel 6.10 to 6.18)
Kernels 6.10 to 6.18 still have similar functionality, but the paths are a little different.
This is untested, and is based on this patch set.
# Check your available GGTT size, this will be the amount you divide between your VFs
sudo cat /sys/kernel/debug/dri/0/gt0/pf/ggtt_available
# Set the amount of GGTT to allocate in bytes (e.g. 512MiB)
echo 536870912 | sudo tee /sys/kernel/debug/dri/0/gt0/vf*/ggtt_quota
# Set the amount of VRAM you want to allocate in bytes (e.g. 2GiB)
echo 2147483648 | sudo tee /sys/kernel/debug/dri/0/gt0/vf*/lmem_quota
# Set the amount of GuC contexts and doorbells to allocate
echo 8192 | sudo tee /sys/kernel/debug/dri/0/gt0/vf*/contexts_quota
echo 32 | sudo tee /sys/kernel/debug/dri/0/gt0/vf*/doorbells_quota
echo 8192 | sudo tee /sys/kernel/debug/dri/0/gt1/vf*/contexts_quota
echo 32 | sudo tee /sys/kernel/debug/dri/0/gt1/vf*/doorbells_quota
# Enable the VFs
echo 7 | sudo tee /sys/bus/pci/drivers/xe/0000\:??\:??.0/sriov_numvfs
Alchemist cards (i915 driver)
The old i915 driver also has similar functionality, refer to Intel’s documentation for more information. They even provide a script!
Notes
- If you are using a root shell, you can replace
| sudo teewith>. - systemd-tmpfiles is a useful way to set these values on boot.
- Code 43 in Windows may mean something isn’t allocated correctly. Check
dmesgto make sureGGTT,LMEM, andGT0>1GuC contexts/doorbells are all provisioned correctly. Try a different method if you are having issues.
Thanks for reading!
Steve.
Comments
Reply