lkml.org 
[lkml]   [2016]   [Nov]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: Enabling peer to peer device transactions for PCIe devices
    Hey,

    On 22/11/16 11:59 AM, Serguei Sagalovitch wrote:
    > - How well we will be able to handle case when we need to "move"/"evict"
    > memory/data to the new location so CPU pointer should point to the
    > new physical location/address
    > (and may be not in PCI device memory at all)?

    IMO any memory that has been registered for a P2P transaction should be
    locked from being evicted. So if there's a get_user_pages call it needs
    to be pinned until the put_page. The main issue being with the RDMA
    case: handling an eviction when a chunk of memory has been registered as
    an MR would be very tricky. The MR may be relied upon by another host
    and the kernel would have to inform user-space the MR was invalid then
    user-space would have to tell the remote application. This seems like a
    lot of burden to place on applications and may be subject to timing
    issues. Either that or all RDMA applications need to be written with the
    assumption that their target memory could go away at any time.

    More generally, if you tell one PCI device to do a DMA transfer to
    another PCI device's BAR space, and the target memory gets evicted then
    DMA transaction needs to be aborted which means every driver doing the
    transfer would need special support for this. If the memory can be
    relied on to not be evicted than existing drivers should work unmodified
    (ie O_DIRECT to/from an NVMe card would just work).

    I feel the better approach is to pin memory subject to P2P transactions
    as is typically done with DMA transfers to main memory.

    Logan

    \
     
     \ /
      Last update: 2016-11-23 18:14    [W:3.171 / U:0.028 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site