Messages in this thread | | | From | Oded Gabbay <> | Date | Tue, 11 Jun 2019 20:22:43 +0300 | Subject | Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9 |
| |
On Tue, Jun 11, 2019 at 8:03 PM Oded Gabbay <oded.gabbay@gmail.com> wrote: > > On Tue, Jun 11, 2019 at 6:26 PM Greg KH <gregkh@linuxfoundation.org> wrote: > > > > On Tue, Jun 11, 2019 at 08:17:53AM -0700, Christoph Hellwig wrote: > > > On Tue, Jun 11, 2019 at 11:58:57AM +0200, Greg KH wrote: > > > > That feels like a big hack. ppc doesn't have any "what arch am I > > > > running on?" runtime call? Did you ask on the ppc64 mailing list? I'm > > > > ok to take this for now, but odds are you need a better fix for this > > > > sometime... > > > > > > That isn't the worst part of it. The whole idea of checking what I'm > > > running to set a dma mask just doesn't make any sense at all. > > > > Oded, I thought I asked if there was a dma call you should be making to > > keep this type of check from being needed. What happened to that? As > > Christoph points out, none of this should be needed, which is what I > > thought I originally said :) > > > > thanks, > > > > greg k-h > > I'm sorry, but it seems I can't explain what's my problem because you > and Christoph keep mentioning the pci_set_dma_mask() but it doesn't > help me. > I'll try again to explain. > > The main problem specifically for Goya device, is that I can't call > this function with *the same parameter* for POWER9 and x86-64, because > x86-64 supports dma mask of 48-bits while POWER9 supports only 32-bits > or 64-bits. > > The main limitation in my Goya device is that it can generate PCI > outbound transactions with addresses from 0 to (2^50 - 1). > That's why when we first integrated it in x86-64, we used a DMA mask > of 48-bits, by calling pci_set_dma_mask(pdev, 48). That way, the > kernel ensures me that all the DMA addresses are from 0 to (2^48 - 1), > and that address range is accessible by my device. > > If for some reason, the x86-64 machine doesn't support 48-bits, the > standard fallback code in ALL the drivers I have seen is to set the > DMA mask to 32-bits. And that's how my current driver's code is > written. > > Now, when I tried to integrate Goya into a POWER9 machine, I got a > reject from the call to pci_set_dma_mask(pdev, 48). The standard code, > as I wrote above, is to call the same function with 32-bits. That > works BUT it is not practical, as our applications require much more > memory mapped then 32-bits. In addition, once you add more cards which > are all mapped to the same range, it is simply not usable at all. > > Therefore, I consulted with POWER people and they told me I can call > to pci_set_dma_mask with the mask as 64, but I must make sure that ALL > outbound transactions from Goya will be with bit 59 set in the > address. > I can achieve that with a dedicated configuration I make in Goya's > PCIe controller. That's what I did and that works. > > So, to summarize: > If I call pci_set_dma_mask with 48, then it fails on POWER9. However, > in runtime, I don't know if its POWER9 or not, so upon failure I will > call it again with 32, which makes our device pretty much unusable. > If I call pci_set_dma_mask with 64, and do the dedicated configuration > in Goya's PCIe controller, then it won't work on x86-64, because bit > 59 will be set and the host won't like it (I checked it). In addition, > I might get addresses above 50 bits, which my device can't generate. > > I hope this makes things more clear. Now, please explain to me how I > can call pci_set_dma_mask without any regard to whether I run on > x86-64 or POWER9, considering what I wrote above ? > > Thanks, > Oded
Adding ppc mailing list.
Oded
| |