Messages in this thread | | | Date | Thu, 10 Aug 2000 16:30:09 +0100 (BST) | From | Corin Hartland-Swann <> | Subject | Degrading disk read performance under 2.2.16 |
| |
Hi there,
Firstly, IANAKH (I am not a kernel-hacker), so please CC: everything to linux-raid. And be gentle with me :)
I have been experimenting with RAID for some time, and found that the big read-access slowdown between kernels 2.2.15 and 2.2.16 on RAID-1 mirrors was because of the raw disk accesses (on IDE) rather than a RAID problem. I benchmarked single IDE disk performance on the following setup:
Intel 810E Chipset Motherboard (CA810EAL), Pentium III-667, 32M RAM, Maxtor DiamondMax Plus 40 40.9GB UDMA66 Disk, Model 54098U8
I have attached the (edited) kernel config I used for all 2.2 kernels FYI.
I used tiotest to benchmark, using a file size of 256MB, block size of 4K, and with 1, 2, 4, 16, 32 threads. The performance starts to get hit as soon as you have multiple threads reading simultaneously, and degrades much more quickly on 2.2.16. Strangely, writes do not degrade.
Size is MB, BlkSz is Bytes, Read and Write are MB/sec, Seeks are Seeks/sec
==> 2.2.15 <== Dir Size BlkSz Thr# Read (CPU%) Write (CPU%) Seeks (CPU%) ----- ------ ------- ---- ------------- -------------- -------------- /mnt/ 256 4096 1 27.1371 10.3% 26.7979 23.0% 146.187 0.95% /mnt/ 256 4096 2 27.1219 10.7% 26.6606 23.2% 142.233 0.60% /mnt/ 256 4096 4 26.9915 10.6% 26.4289 22.9% 142.789 0.50% /mnt/ 256 4096 16 26.4320 10.5% 26.1310 23.0% 147.424 0.52% /mnt/ 256 4096 32 25.3407 10.1% 25.6822 22.7% 150.750 0.57%
==> 2.2.16 <== Dir Size BlkSz Thr# Read (CPU%) Write (CPU%) Seeks (CPU%) ----- ------ ------- ---- ------------- -------------- -------------- /mnt/ 256 4096 1 27.0948 10.5% 25.9272 22.5% 147.264 0.83% /mnt/ 256 4096 2 20.5753 8.42% 25.9316 22.7% 143.113 0.56% /mnt/ 256 4096 4 13.0397 6.09% 25.7723 22.7% 143.619 0.51% /mnt/ 256 4096 16 9.38573 5.40% 25.6285 23.1% 147.815 0.51% /mnt/ 256 4096 32 6.96358 5.90% 25.2889 22.9% 151.182 0.57%
The performance of 2.2.17-pre9 is basically exactly the same as 2.2.16, except that it died with do_vm_free_pages() when trying to run it with 64 threads.
I had a look in drivers/block/ide.c, and the only big change to seems to be adding a new element 'expiry' to ide_hwgroup_t. ide_set_handler() has been changed to take it as an extra argument, however, the only place that I can find calls to ide_set_handler() with a non-NULL expiry is in ide-cd.c, (I haven't tested CD-ROMs to see if there's a slow-down there as well). All calls to ide_set_handler() in ide.c and ide-disk.c specify expiry as NULL.
The only place where hwgroup->expiry is set is in ide_set_handler (always passed NULL by the HD routines). The only place that it is examined is in ide_timer_expiry() which should not execute any more code if expiry is NULL.
I also tested 2.4.0-test5, and performance there sucks with one thread, but doesn't degrade any more quickly than 2.2.15
==> 2.4.0-test5 <==
Dir Size BlkSz Thr# Read (CPU%) Write (CPU%) Seeks (CPU%) ----- ------ ------- ---- ------------- -------------- -------------- /mnt/ 256 4096 1 8.15398 49.6% 7.64104 71.2% 125.675 5.90% /mnt/ 256 4096 2 8.01528 56.1% 7.63085 69.8% 123.076 5.59% /mnt/ 256 4096 4 7.79720 61.6% 7.62060 68.1% 125.437 5.75% /mnt/ 256 4096 16 7.64617 65.2% 7.60466 62.5% 129.431 5.83% /mnt/ 256 4096 32 7.60580 67.6% 7.58702 51.7% 132.791 5.95%
I haven't tried any of this with SCSI disks, so the problem may well lie in VFS or the VM subsystem (I know there are problems there as well). I don't know anything about those, so I guess I should just leave it to the professionals now :)
Thanks,
Corin
/------------------------+-------------------------------------\ | Corin Hartland-Swann | Direct: +44 (0) 20 7544 4676 | | Commerce Internet Ltd | Mobile: +44 (0) 79 5854 0027 | | 22 Cavendish Buildings | Tel: +44 (0) 20 7491 2000 | | Gilbert Street | Fax: +44 (0) 20 7491 2010 | | Mayfair | Web: http://www.commerce.uk.net/ | | London W1K 5HJ | E-Mail: cdhs@commerce.uk.net | \------------------------+-------------------------------------/ CONFIG_EXPERIMENTAL=y CONFIG_M686=y CONFIG_X86_WP_WORKS_OK=y CONFIG_X86_INVLPG=y CONFIG_X86_BSWAP=y CONFIG_X86_POPAD_OK=y CONFIG_X86_TSC=y CONFIG_X86_GOOD_APIC=y CONFIG_1GB=y CONFIG_MTRR=y CONFIG_MODULES=y CONFIG_MODVERSIONS=y CONFIG_KMOD=y CONFIG_NET=y CONFIG_PCI=y CONFIG_PCI_GOANY=y CONFIG_PCI_BIOS=y CONFIG_PCI_DIRECT=y CONFIG_PCI_QUIRKS=y CONFIG_PCI_OLD_PROC=y CONFIG_SYSVIPC=y CONFIG_SYSCTL=y CONFIG_BINFMT_AOUT=y CONFIG_BINFMT_ELF=y CONFIG_BINFMT_MISC=y CONFIG_BLK_DEV_FD=y CONFIG_BLK_DEV_IDE=y CONFIG_BLK_DEV_IDEDISK=y CONFIG_BLK_DEV_IDECD=y CONFIG_BLK_DEV_CMD640=y CONFIG_BLK_DEV_RZ1000=y CONFIG_BLK_DEV_IDEPCI=y CONFIG_BLK_DEV_IDEDMA=y CONFIG_IDEDMA_AUTO=y CONFIG_BLK_DEV_LOOP=y CONFIG_BLK_DEV_MD=y CONFIG_MD_LINEAR=y CONFIG_MD_STRIPED=y CONFIG_MD_MIRRORING=y CONFIG_MD_RAID5=y CONFIG_MD_BOOT=y CONFIG_PARIDE_PARPORT=y CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_INET=y CONFIG_IP_ALIAS=y CONFIG_SKB_LARGE=y CONFIG_SCSI=y CONFIG_BLK_DEV_SD=y CONFIG_CHR_DEV_ST=y CONFIG_BLK_DEV_SR=y CONFIG_SCSI_CONSTANTS=y CONFIG_SCSI_AIC7XXX=y CONFIG_AIC7XXX_CMDS_PER_DEVICE=8 CONFIG_AIC7XXX_RESET_DELAY=5 CONFIG_SCSI_ADVANSYS=y CONFIG_SCSI_INIA100=y CONFIG_NETDEVICES=y CONFIG_DUMMY=y CONFIG_NET_ETHERNET=y CONFIG_NET_VENDOR_3COM=y CONFIG_VORTEX=y CONFIG_NET_EISA=y CONFIG_EEXPRESS_PRO100=y CONFIG_VIA_RHINE=y CONFIG_VT=y CONFIG_VT_CONSOLE=y CONFIG_SERIAL=y CONFIG_SERIAL_CONSOLE=y CONFIG_UNIX98_PTYS=y CONFIG_UNIX98_PTY_COUNT=2048 CONFIG_MOUSE=y CONFIG_PSMOUSE=y CONFIG_82C710_MOUSE=y CONFIG_RTC=y CONFIG_AUTOFS_FS=y CONFIG_ISO9660_FS=y CONFIG_PROC_FS=y CONFIG_DEVPTS_FS=y CONFIG_EXT2_FS=y CONFIG_NFS_FS=y CONFIG_SUNRPC=y CONFIG_LOCKD=y CONFIG_VGA_CONSOLE=y CONFIG_VIDEO_SELECT=y
| |