Messages in this thread | | | Date | Wed, 25 Oct 2006 11:15:54 -0600 | From | Jason Gunthorpe <> | Subject | Re: [openib-general] Ordering between PCI config space writes and MMIO reads? |
| |
On Wed, Oct 25, 2006 at 08:18:59AM -0600, Matthew Wilcox wrote:
> PCI-PCI bridges are allowed to do it. If you look in table E-1 of PCI > 2.3, or table 8-3 of PCI-X 2.0, you'll see that a Posted Memory Write > can pass a Delayed Write Request (or in PCI-X, a Memory Write can pass a > Split Write Request).
Carefull here.. Only MMIO writes are of the posted variety. Non posted transactions (config write, IO write, all reads) are not ever allowed to pass a posted write, but they can be re-ordered.
Table 8-3 shows that Split Write vs Split Read are all Y/N meaning they could be re-ordered with respect to each other.
The problem with mthca is exactly this, although the operations were issued in-order by the bridges the end device (mthca) is free to complete them in any order, and it choose to complete the MMIO read before the config write.
> In most PCI-X implementations, Split Requests are managed in separate > buffers from Split Completions, so Split Requests naturally pass Split > Completions. However, no deadlocks occur if Split Completions block > Split Requests.
Again, this is only for posted writes. All these rules are designed to prevent the bus from deadlocking due to buffer starvation under certain situations. [Basically split completions and posted writes are given a seperate queue that can advance if the request queue is stalled. Otherwise you can deadlock a bridge]
> So all this code that checks to see if a write had an effect is unsafe. > I'm a little perturbed by this. It means the only way to reliably
MMIO based code that does this is correct and reliable.. PIO code that does this is only safe if the platform is waiting for the PIO write completion before starting the PIO read.
PCI-X (pg 80) says this about non-posted transaction ordering requirements: As in convention PCI, if a requester requires one non-posted transaction to complete before another, it must not initiate the second transaction until the first one compeltes.
IMHO, all sane hardware implementations of config ops and PIO should block the host bridge until the completion is generated by the end device..I'm sure that most x86 platforms do this. (For instance I've observed this kind of behavior with a Hyper Transport probe on Opterons)
This is more than just worrying about ordering, it is about how to engage a platform specific way to know that the completion has been generated. The person who suggested polling the PIO_OUTSTANDING register on SN2 seems to have the right idea (if that counts all pending non-posted operations, not just PIO ones) :|
The risk of re-ordering is probably not so much in the bridges since that would be a fairly strange thing to do - but it is very likely in end-devices. This is especially true if the accesses are to different internal resources!
Jason - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |