[lkml]   [2006]   [Oct]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [openib-general] Ordering between PCI config space writes and MMIO reads?
    On Wed, Oct 25, 2006 at 08:18:59AM -0600, Matthew Wilcox wrote:

    > PCI-PCI bridges are allowed to do it. If you look in table E-1 of PCI
    > 2.3, or table 8-3 of PCI-X 2.0, you'll see that a Posted Memory Write
    > can pass a Delayed Write Request (or in PCI-X, a Memory Write can pass a
    > Split Write Request).

    Carefull here.. Only MMIO writes are of the posted variety. Non posted
    transactions (config write, IO write, all reads) are not ever allowed
    to pass a posted write, but they can be re-ordered.

    Table 8-3 shows that Split Write vs Split Read are all Y/N
    meaning they could be re-ordered with respect to each other.

    The problem with mthca is exactly this, although the operations were
    issued in-order by the bridges the end device (mthca) is free to
    complete them in any order, and it choose to complete the MMIO read
    before the config write.

    > In most PCI-X implementations, Split Requests are managed in separate
    > buffers from Split Completions, so Split Requests naturally pass Split
    > Completions. However, no deadlocks occur if Split Completions block
    > Split Requests.

    Again, this is only for posted writes. All these rules are designed to
    prevent the bus from deadlocking due to buffer starvation under
    certain situations. [Basically split completions and posted writes are
    given a seperate queue that can advance if the request queue is
    stalled. Otherwise you can deadlock a bridge]

    > So all this code that checks to see if a write had an effect is unsafe.
    > I'm a little perturbed by this. It means the only way to reliably

    MMIO based code that does this is correct and reliable.. PIO code that
    does this is only safe if the platform is waiting for the PIO write
    completion before starting the PIO read.

    PCI-X (pg 80) says this about non-posted transaction ordering requirements:
    As in convention PCI, if a requester requires one non-posted
    transaction to complete before another, it must not initiate the
    second transaction until the first one compeltes.

    IMHO, all sane hardware implementations of config ops and PIO should
    block the host bridge until the completion is generated by the end
    device..I'm sure that most x86 platforms do this. (For instance I've
    observed this kind of behavior with a Hyper Transport probe on

    This is more than just worrying about ordering, it is about how to
    engage a platform specific way to know that the completion has been
    generated. The person who suggested polling the PIO_OUTSTANDING
    register on SN2 seems to have the right idea (if that counts all
    pending non-posted operations, not just PIO ones) :|

    The risk of re-ordering is probably not so much in the bridges since
    that would be a fairly strange thing to do - but it is very likely in
    end-devices. This is especially true if the accesses are to different
    internal resources!

    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2006-10-25 19:19    [W:0.023 / U:2.676 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site