Messages in this thread | | | Date | Tue, 30 Aug 2005 20:06:21 +0000 (GMT) | From | Holger Kiehl <> | Subject | Re: Where is the performance bottleneck? |
| |
On Mon, 29 Aug 2005, Vojtech Pavlik wrote:
> On Mon, Aug 29, 2005 at 06:20:56PM +0000, Holger Kiehl wrote: >> Hello >> >> I have a system with the following setup: >> >> Board is Tyan S4882 with AMD 8131 Chipset >> 4 Opterons 848 (2.2GHz) >> 8 GB DDR400 Ram (2GB for each CPU) >> 1 onboard Symbios Logic 53c1030 dual channel U320 controller >> 2 SATA disks put together as a SW Raid1 for system, swap and spares >> 8 SCSI U320 (15000 rpm) disks where 4 disks (sdc, sdd, sde, sdf) >> are on one channel and the other four (sdg, sdh, sdi, sdj) on >> the other channel. >> >> The U320 SCSI controller has a 64 bit PCI-X bus for itself, there is >> no other device on that bus. Unfortunatly I was unable to determine at >> what speed it is running, here the output from lspci -vv: > >> How does one determine the PCI-X bus speed? > > Usually only the card (in your case the Symbios SCSI controller) can > tell. If it does, it'll be most likely in 'dmesg'. > There is nothing in dmesg:
Fusion MPT base driver 3.01.20 Copyright (c) 1999-2004 LSI Logic Corporation ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 24 (level, low) -> IRQ 217 mptbase: Initiating ioc0 bringup ioc0: 53C1030: Capabilities={Initiator,Target} ACPI: PCI Interrupt 0000:02:04.1[B] -> GSI 25 (level, low) -> IRQ 225 mptbase: Initiating ioc1 bringup ioc1: 53C1030: Capabilities={Initiator,Target} Fusion MPT SCSI Host driver 3.01.20
>> Anyway, I thought with this system I would get theoretically 640 MB/s using >> both channels. > > You can never use the full theoretical bandwidth of the channel for > data. A lot of overhead remains for other signalling. Similarly for PCI. > >> I tested several software raid setups to get the best possible write >> speeds for this system. But testing shows that the absolute maximum I >> can reach with software raid is only approx. 270 MB/s for writting. >> Which is very disappointing. > > I'd expect somewhat better (in the 300-400 MB/s range), but this is not > too bad. > > To find where the bottleneck is, I'd suggest trying without the > filesystem at all, and just filling a large part of the block device > using the 'dd' command. > > Also, trying without the RAID, and just running 4 (and 8) concurrent > dd's to the separate drives could show whether it's the RAID that's > slowing things down. > Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
Here the results:
Each disk alone /dev/sdc1 59.094636 MB/s /dev/sdd1 58.686592 MB/s /dev/sde1 55.282807 MB/s /dev/sdf1 62.271240 MB/s /dev/sdg1 60.872891 MB/s /dev/sdh1 62.252781 MB/s /dev/sdi1 59.145637 MB/s /dev/sdj1 60.921119 MB/s
sdc + sdd in parallel (2 disks on same channel) /dev/sdc1 42.512287 MB/s /dev/sdd1 43.118483 MB/s
sdc + sdg in parallel (2 disks on different channels) /dev/sdc1 42.938186 MB/s /dev/sdg1 43.934779 MB/s
sdc + sdd + sde in parallel (3 disks on same channel) /dev/sdc1 35.043501 MB/s /dev/sdd1 35.686878 MB/s /dev/sde1 34.580457 MB/s
Similar results for three disks (sdg + sdh + sdi) on the other channel /dev/sdg1 36.381137 MB/s /dev/sdh1 37.541758 MB/s /dev/sdi1 35.834920 MB/s
sdc + sdd + sde + sdf in parallel (4 disks on same channel) /dev/sdc1 31.432914 MB/s /dev/sdd1 32.058752 MB/s /dev/sde1 31.393455 MB/s /dev/sdf1 33.208165 MB/s
And here for the four disks on the other channel /dev/sdg1 31.873028 MB/s /dev/sdh1 33.277193 MB/s /dev/sdi1 31.910000 MB/s /dev/sdj1 32.626744 MB/s
All 8 disks in parallel /dev/sdc1 24.120545 MB/s /dev/sdd1 24.419801 MB/s /dev/sde1 24.296588 MB/s /dev/sdf1 25.609548 MB/s /dev/sdg1 24.572617 MB/s /dev/sdh1 25.552590 MB/s /dev/sdi1 24.575616 MB/s /dev/sdj1 25.124165 MB/s
So from these results, I may assume that md is not the cause of the problem.
What comes as a big surprise is that I loose 25% performance with only two disks and each hanging on its own channel!
Is this normal? I wonder if other people have the same problem with other controllers or the same.
What can I do next to find out if this is a kernel, driver or hardware problem?
Thanks, Holger
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |