lkml.org 
[lkml]   [2007]   [Dec]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [ofa-general] iommu dma mapping alignment requirements
    From
    Date

    On Thu, 2007-12-20 at 11:14 -0600, Steve Wise wrote:
    > Hey Roland (and any iommu/ppc/dma experts out there):
    >
    > I'm debugging a data corruption issue that happens on PPC64 systems
    > running rdma on kernels where the iommu page size is 4KB yet the host
    > page size is 64KB. This "feature" was added to the PPC64 code recently,
    > and is in kernel.org from 2.6.23. So if the kernel is built with a 4KB
    > page size, no problems. If the kernel is prior to 2.6.23 then 64KB page
    > configs work too. Its just a problem when the iommu page size != host
    > page size.
    >
    > It appears that my problem boils down to a single host page of memory
    > that is mapped for dma, and the dma address returned by dma_map_sg() is
    > _not_ 64KB aligned. Here is an example:
    >
    > app registers va 0x000000002d9a3000 len 12288
    > ib_umem_get() creates and maps a umem and chunk that looks like (dumping
    > state from a registered user memory region):
    >
    > > umem len 12288 off 12288 pgsz 65536 shift 16
    > > chunk 0: nmap 1 nents 1
    > > sglist[0] page 0xc000000000930b08 off 0 len 65536 dma_addr 000000005bff4000 dma_len 65536
    > >
    >
    > So the kernel maps 1 full page for this MR. But note that the dma
    > address is 000000005bff4000 which is 4KB aligned, not 64KB aligned. I
    > think this is causing grief to the RDMA HW.
    >
    > My first question is: Is there an assumption or requirement in linux
    > that dma_addressess should have the same alignment as the host address
    > they are mapped to? IE the rdma core is mapping the entire 64KB page,
    > but the mapping doesn't begin on a 64KB page boundary.
    >
    > If this mapping is considered valid, then perhaps the rdma hw is at
    > fault here. But I'm wondering if this is an PPC/iommu bug.
    >
    > BTW: Here is what the Memory Region looks like to the HW:
    >
    > > TPT entry: stag idx 0x2e800 key 0xff state VAL type NSMR pdid 0x2
    > > perms RW rem_inv_dis 0 addr_type VATO
    > > bind_enable 1 pg_size 65536 qpid 0x0 pbl_addr 0x003c67c0
    > > len 12288 va 000000002d9a3000 bind_cnt 0
    > > PBL: 000000005bff4000
    >
    >
    >
    > Any thoughts?

    The Ammasso certainly works this way. If you tell it the page size is
    64KB, it will ignore bits in the page address that encode 0-65535.

    >
    > Steve.
    >
    >
    > _______________________________________________
    > general mailing list
    > general@lists.openfabrics.org
    > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
    >
    > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general



    \
     
     \ /
      Last update: 2007-12-20 18:27    [W:0.026 / U:0.112 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site