Messages in this thread |  | | From | "Boone, Max" <> | | Subject | Re: [PATCH RFC] net/mlx5: check whether VFs are assigned before disabling SR-IOV | | Date | Wed, 29 Apr 2026 13:29:57 +0000 |
| |
> On Apr 29, 2026, at 2:38 PM, Jason Gunthorpe <jgg@ziepe.ca> wrote: > > !-------------------------------------------------------------------| > This Message Is From an External Sender > This message came from outside your organization. > |-------------------------------------------------------------------! > > On Tue, Apr 28, 2026 at 08:04:14PM +0200, Max Boone via B4 Relay wrote: >> From: Max Boone <mboone@akamai.com> >> >> When MLX5 cards are passed through to a VM, disabling SR-IOV by >> setting the sriov_numvfs to 0 will render the machine unstable. > > What? How does that happen?
Unstable is maybe a bit confusing phrasing on my part, “locks up” might be a better description?
In short: - Enable by setting sriov_numvfs to positive - vfio-pci passthrough to QEMU (or other process) - Disable by setting sriov_numvfs to zero - QEMU processes freeze, shell that was writing to sysfs freezes - SIGKILL doesn’t seem to have much effect, shutdown never completes
Python script to reproduce without QEMU: - https://github.com/akamaxb/repro-vfio-sriov-removal/blob/main/vfio-sriov-bind.py
Does: 1. Require sriov_numvfs == 0 on the PF (report any existing users and exit if not) 2. Add one SR-IOV VF 3. Bind the VF to vfio-pci via driver_override + drivers_probe 4. Open VFIO container + group, get device fd 5. Create a KVM VM (registers an MMU notifier — required to trigger the race) 6. Hold and wait for user input
To trigger the bug while the script is waiting, in another terminal: echo 0 > /sys/bus/pci/devices/<pf_device>/sriov_numvfs
On the vfio-pci end of it all, it prints these two lines to dmesg before it hangs: - https://elixir.bootlin.com/linux/v7.0.1/source/drivers/vfio/pci/vfio_pci_core.c#L1826 - https://elixir.bootlin.com/linux/v7.0.1/source/drivers/vfio/vfio_main.c#L421
>> -void mlx5_sriov_disable(struct pci_dev *pdev, bool num_vf_change) >> +int mlx5_sriov_disable(struct pci_dev *pdev, bool num_vf_change) >> { >> struct mlx5_core_dev *dev = pci_get_drvdata(pdev); >> struct devlink *devlink = priv_to_devlink(dev); >> int num_vfs = pci_num_vf(dev->pdev); >> >> + if (pci_vfs_assigned(dev->pdev)) { >> + mlx5_core_warn(dev, "can't disable sriov, VFs are assigned\n"); >> + return -EPERM; >> + } > > *barf* WTF did this come from?
Hahaha, take your pick: - https://elixir.bootlin.com/linux/v7.0.1/C/ident/pci_vfs_assigned
I followed the sysfs sriov_numvfs op for a couple drivers and saw that ixgbe (and others) had it plumbed in, so presumed (sorry) that this would fix it / was an obvious omission if the rest is doing it. My bad for cargo culting an artifact from Xen.
> Grep says only Xen makes this true, so this is all working around some > Xen brokenness in their "assignment" ?
Yeap, I see, looks like it.
> If people care about Xen pci_is_dev_assigned() should be be purged and > pciback should be fixed to not "make the machine unstable" when it is > removed during a VF teardown. > > Or at the very least this nasty Xen intrustion should be placed in the > PCI core code and removed from the drivers. > > Also, no, you can't fail mlx5_sriov_disable() it is called during > driver remove and cannot fail in that flow.
Check. I can do some further digging and build a kernel with lockdep to try and find what it is hanging on specifically. Unless something pops to mind?
> > Jason
[unhandled content-type:application/pkcs7-signature]
|  |