Messages in this thread | | | Date | Thu, 5 Jan 2017 15:42:15 -0700 | From | Jason Gunthorpe <> | Subject | Re: Enabling peer to peer device transactions for PCIe devices |
| |
On Thu, Jan 05, 2017 at 03:19:36PM -0500, Jerome Glisse wrote:
> > Always having a VMA changes the discussion - the question is how to > > create a VMA that reprensents IO device memory, and how do DMA > > consumers extract the correct information from that VMA to pass to the > > kernel DMA API so it can setup peer-peer DMA. > > Well my point is that it can't be. In HMM case inside a single VMA > you [..]
> In the GPUDirect case the idea is that you have a specific device vma > that you map for peer to peer.
[..]
I still don't understand what you driving at - you've said in both cases a user VMA exists.
From my perspective in RDMA, all I want is a core kernel flow to convert a '__user *' into a scatter list of DMA addresses, that works no matter what is backing that VMA, be it HMM, a 'hidden' GPU object, or struct page memory.
A '__user *' pointer is the only way to setup a RDMA MR, and I see no reason to have another API at this time.
The details of how to translate to a scatter list are a MM subject, and the MM folks need to get
I just don't care if that routine works at a page level, or a whole VMA level, or some combination of both, that is up to the MM team to figure out :)
> a page level. Expectation here is that the GPU userspace expose a special > API to allow RDMA to directly happen on GPU object allocated through > GPU specific API (ie it is not regular memory and it is not accessible > by CPU).
So, how do you identify these GPU objects? How do you expect RDMA convert them to scatter lists? How will ODP work?
> > We have MMU notifiers to handle this today in RDMA. Async RDMA MR > > Invalidate like you see in the above out of tree patches is totally > > crazy and shouldn't be in mainline. Use ODP capable RDMA hardware. > > Well there is still a large base of hardware that do not have such > feature and some people would like to be able to keep using those.
Hopefully someone will figure out how to do that without the crazy async MR invalidation.
Jason
| |