Messages in this thread | | | From | "Fab Tillier" <> | Subject | RE: PCI bandwidth measurement methods | Date | Tue, 15 Jun 2004 14:58:24 -0700 |
| |
> From: Arthur Perry [mailto:kernel@linuxfarms.com] > Sent: Tuesday, June 15, 2004 1:56 PM > > Hello, > > I have a question about methods of measuring PCI-X bandwidth. > I was wondering if anybody has any ideas. > What I am looking for is an "industry standard" method, if one exists. > At this point, such software does not exist in SPEC. > Different HBA cards such as Myricom and Mellanox (Myrinet and Infiniband > hosts) have utilitites to test your PCI bandwidth.. > But what do you do when they come up with completely different numbers?
There are many variables that will determine peak bandwidth for an I/O adapter. The size of the I/O request as well as the number of I/O requests outstanding affect bandwidth. The PCI commands used by the adapters to transfer the memory (burst length, etc) will change the amount of overhead for the different devices too.
Unless mistaken, Myrinet PCI-X adapters operate at 2.5Gb full duplex per port, while the Mellanox 4x InfiniBand HCA operates at 10Gb full duplex per port. From the HW characteristics, you should be able to get 4x the bandwidth out of the IB HCA (assuming linear scale-up and comparable HW operation), so that might explain your bandwidth discrepancy between the different HW.
Lastly, a PCI analyzer can be useful in seeing exactly what commands are used and how large a burst the different HW adapters are capable of, but that moves you out of pure SW benchmarking.
> > What I need is an unbiased, objective PCI-X bandwidth test. > It's all about performance in Linux actually... So I can't say "completely > objective"..
You don't need a single test. What you need is to control what each test does. Same size I/O, same number of outstanding requests, etc. If you have that, you should be able to compare.
> > I can write software that will do this, possibly by performing PCI > writes to scratch memory area on several different HBAs..
PCI writes to HBA attached memory may not use the same PCI commands as I/O DMA, and thus might lead to totally different results.
> Or maybe adding a counter into the Linux PCI device driver to verify the > bandwidth claims that Mellanox and Myricom's tools report. > But before I re-invent the wheel, I was wondering if anybody knew of > anything that is already out there.
Things like MPI benchmarks should do the trick, since only the underlying HW provider changes. I'm assuming you can get MPI on top of Myrinet.
> If I were to write something, a whole software validation process would > have to be performed on top of the actual work. > If anybody knows of something that already exists and has been in use, it > would be ideal.
Give MPI benchmarks a try.
Good luck,
- Fab
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |