Messages in this thread | | | From | "Oliver O'Halloran" <> | Date | Wed, 12 Jun 2019 15:45:17 +1000 | Subject | Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9 |
| |
On Wed, Jun 12, 2019 at 8:54 AM Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote: > > On Tue, 2019-06-11 at 20:22 +0300, Oded Gabbay wrote: > > > > > So, to summarize: > > > If I call pci_set_dma_mask with 48, then it fails on POWER9. However, > > > in runtime, I don't know if its POWER9 or not, so upon failure I will > > > call it again with 32, which makes our device pretty much unusable. > > > If I call pci_set_dma_mask with 64, and do the dedicated configuration > > > in Goya's PCIe controller, then it won't work on x86-64, because bit > > > 59 will be set and the host won't like it (I checked it). In addition, > > > I might get addresses above 50 bits, which my device can't generate. > > > > > > I hope this makes things more clear. Now, please explain to me how I > > > can call pci_set_dma_mask without any regard to whether I run on > > > x86-64 or POWER9, considering what I wrote above ? > > > > > > Thanks, > > > Oded > > > > Adding ppc mailing list. > > You can't. Your device is broken. Devices that don't support DMAing to > the full 64-bit deserve to be added to the trash pile. > > As a result, getting it to work will require hacks. Some GPUs have > similar issues and require similar hacks, it's unfortunate. > > Added a couple of guys on CC who might be able to help get those hacks > right.
> It's still very fishy .. the idea is to detect the case where setting a > 64-bit mask will give your system memory mapped at a fixed high address > (1 << 59 in our case) and program that in your chip in the "Fixed high > bits" register that you seem to have (also make sure it doesn't affect > MSIs or it will break them).
Judging from the patch (https://lkml.org/lkml/2019/6/11/59) this is what they're doing.
Also, are you sure about the MSI thing? The IODA3 spec says the only important bits for a 64bit MSI are bits 61:60 (to hit the window) and the lower bits that determine what IVE to use. Everything in between is ignored so ORing in bit 59 shouldn't break anything.
> This will only work as long as all of the system memory can be > addressed at an offset from that fixed address that itself fits your > device addressing capabilities (50 bits in this case). It may or may > not be the case but there's no way to check since the DMA mask logic > won't really apply. > > You might want to consider fixing your HW in the next iteration... This > is going to bite you when x86 increases the max physical memory for > example, or on other architectures.
Yes, do this. The easiest way to avoid this sort of wierd hack is to just design the PCIe interface to the spec in the first place.
| |