Messages in this thread | | | Date | Wed, 8 Dec 2021 17:12:23 -0700 | Subject | Re: [PATCH 4/4] dmaengine: idxd: Use DMA API for in-kernel DMA with PASID | From | Dave Jiang <> |
| |
On 12/8/2021 4:39 PM, Jason Gunthorpe wrote: > On Wed, Dec 08, 2021 at 01:59:45PM -0800, Jacob Pan wrote: >> Hi Jason, >> >> On Wed, 8 Dec 2021 16:30:22 -0400, Jason Gunthorpe <jgg@nvidia.com> wrote: >> >>> On Wed, Dec 08, 2021 at 11:55:16AM -0800, Jacob Pan wrote: >>>> Hi Jason, >>>> >>>> On Wed, 8 Dec 2021 09:13:58 -0400, Jason Gunthorpe <jgg@nvidia.com> >>>> wrote: >>>>>> This patch utilizes iommu_enable_pasid_dma() to enable DSA to >>>>>> perform DMA requests with PASID under the same mapping managed by >>>>>> DMA mapping API. In addition, SVA-related bits for kernel DMA are >>>>>> removed. As a result, DSA users shall use DMA mapping API to obtain >>>>>> DMA handles instead of using kernel virtual addresses. >>>>> Er, shouldn't this be adding dma_map/etc type calls? >>>>> >>>>> You can't really say a driver is using the DMA API without actually >>>>> calling the DMA API.. >>>> The IDXD driver is not aware of addressing mode, it is up to the user of >>>> dmaengine API to prepare the buffer mappings. Here we only set up the >>>> PASID such that it can be picked up during DMA work submission. I >>>> tested with /drivers/dma/dmatest.c which does dma_map_page(), >>>> map_single etc. also tested with other pieces under development. >>> Ignoring the work, doesn't IDXD prepare the DMA queues itself, don't >>> those need the DMA API? >>> >> Do you mean wq completion record address? It is already using DMA API. >> wq->compls = dma_alloc_coherent(dev, wq->compls_size, >> &wq->compls_addr, GFP_KERNEL); >> desc->compl_dma = wq->compls_addr + idxd->data->compl_size * i; > I would have expected something on the queue submission side too?
DSA is different than typical DMA devices in the past. Instead of a software descriptor ring where the device DMA to fetch the descriptors after the software ringing a doorbell or writing a head index, the descriptors are submitted directly to the device via a CPU instruction (i.e. MOVDIR64B or ENQCMD(S)). The CPU takes the KVA of the 64B descriptor and writes to the device atomically. No DMA mapping is necessary in this case.
> >>> the same thing, they do not use the same IOVA's. Did you test this >>> with bypass mode off? >> Yes with dmatest. IOVA is the default, I separated out the SATC patch which >> will put internal accelerators in bypass mode. It can also be verified by >> iommu debugfs dump of DMA PASID (2) and PASID 0 (RIDPASID) are pointing to >> he same default domain. e.g > Well, OK then.. > > Jason
| |