[lkml]   [2012]   [Jun]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: IO_PAGE_FAULTS with igb or igbvf on AMD IOMMU system
    On 06/20/2012 02:48 AM, Joerg Roedel wrote:
    > Hi Alexander,
    > On Tue, Jun 19, 2012 at 11:19:20AM -0700, Alexander Duyck wrote:
    >> Based on the faults it would look like accessing the descriptor rings is
    >> probably triggering the errors. We allocate the descriptor rings using
    >> dma_alloc_coherent so the rings should be mapped correctly.
    > Can this happen before the driver actually allocated the descriptors? As
    > I said, the faults appear before any DMA-API call was made for that
    > device (hence, domain=0x0000, because the domain is assigned on the
    > first call to the DMA-API for a device).
    > Also, I don't see the faults every time. One out of ten times
    > (estimated) there are not faults. Is it possible that this is a race
    > condition, e.g. that the card trys to access its descriptor rings before
    > the driver allocated them (or something like that).
    The descriptor rings are located in system memory so it shouldn't be
    possible for the driver to start accessing anything without requesting
    the address via the DMA-API calls. As I said we use dma_alloc_coherent
    to allocate the descriptor rings so I don't think it is possible for us
    to configure the rings without calling the DMA-API. You might try
    double checking the timing on igbvf driver load versus the notifier
    getting to the IOMMU. I supposed this could be some sort of
    initialization race between the driver and the IOMMU. One way to verify
    that would be to blacklist the VF driver and make sure it is unloaded
    prior to loading the PF driver. Load it manually via insmod after you
    have have created the PCI devices by loading the PF module. This way
    you can guarantee that the igbvf load isn't somehow occuring prior to
    the IOMMU notifier being triggered.

    >> The PF and VF will end up being locked out since they are hung on an
    >> uncompleted DMA transaction. Normally we recommend that PCIe Advanced
    >> Error Reporting be enabled if an IOMMU is enabled so the device can be
    >> reset after triggering a page fault event.
    >> The first thing that pops into my head for possible issues would be that
    >> maybe the VF pci_dev structure or the device structure isn't being
    >> correctly initialized when SR-IOV is enabled on the igb interface. Do
    >> you know if there are any AMD IOMMU specific values on those structures,
    >> such as the domain, that are supposed to be initialized prior to calling
    >> the DMA API calls? If so, have you tried adding debug output to verify
    >> if those values are initialized on a VF prior to bringing up a VF interface?
    > Well, when the device appears in the system the IOMMU driver gets
    > notified about it using the device_change notifiers. It will then
    > allocate all necessary data structures. I also verified that this works
    > correctly while debugging this issue. So I am pretty sure the problem
    > isn't there :)
    You might want to try also adding some code to the dma_alloc_coherent
    function in your kernel to verify that when the VF driver is allocating
    the descriptor rings that the domain info is set-up correctly.

    >> Also have you tried any other SR-IOV capable devices on this system?
    >> That would be a valuable data point because we could then exclude the
    >> SR-IOV code as being a possible cause for the issues if other SR-IOV
    >> devices are working without any issues.
    > I have another SR-IOV device, but that fails to even enable SR-IOV
    > because the BIOS did not let enough MMIO resources left. So I couldn't
    > try it with that device. With the 82576 card enabling SR-IOV works fine
    > but results in the faults from the VF.
    If you are working on a recent kernel you should be able to overcome the
    BIOS issues. I believe there are the options "pci=assign-busses" if the
    BIOS doesn't place enough buses on the bridge to support SR-IOV, and
    there is "pci=realloc" which will reassign the MMIO resources to make
    enough room for VF MMIO bars.



     \ /
      Last update: 2012-06-21 01:21    [W:0.046 / U:14.756 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site