Messages in this thread | | | Date | Mon, 22 May 2006 14:21:48 +0200 (MEST) | From | Jan Engelhardt <> | Subject | Re: XFS write speed drop |
| |
Hi Nathan,
>> I have noticed that after an upgrade from 2.6.16-rcX -> 2.6.17-rc4, writes >> to one (hdc) xfs filesystem have become significantly slower (factor 6 to > >Buffered writes? Direct writes? Sync writes? Log writes?
How would I know? :) "The standard thing". I shuffled a bit in the source code and the word 'order' is quite often around 'barrier', so I thought the 'barrier' option makes the IO scheduler strictly ordered, that is, write log first, then data. (like ext3's data=ordered). Anyhow, just see below
>You're a bit light on details here. > >Can you send the benchmark results themselves please? (as in, >the test(s) you've run that lead you to see 6-8x, and the data >those tests produced). Also, xfs_info output, and maybe list >the device driver(s) involved here too.
Drivers ======= xfs.ko sis5513.ko ide_disk.ko ide_core.ko
xfs_info ======== 14:20 shanghai:/root # xfs_info / meta-data=/dev/hda3 isize=256 agcount=19, agsize=462621 blks = sectsz=512 attr=1 data = bsize=4096 blocks=8586742, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=3614, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=65536 blocks=0, rtextents=0 14:20 shanghai:/root # xfs_info /D meta-data=/dev/hdc2 isize=256 agcount=16, agsize=3821963 blks = sectsz=512 attr=1 data = bsize=4096 blocks=61151408, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=29859, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=65536 blocks=0, rtextents=0 14:20 shanghai:/root # fdisk -l /dev/hda
Disk /dev/hda: 40.0 GB, 40020664320 bytes 255 heads, 63 sectors/track, 4865 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/hda1 1 64 514048+ 82 Linux swap / Solaris /dev/hda2 * 65 588 4209030 c W95 FAT32 (LBA) /dev/hda3 590 4865 34346970 83 Linux 14:21 shanghai:/root # fdisk -l /dev/hdc
Disk /dev/hdc: 251.0 GB, 251000193024 bytes 255 heads, 63 sectors/track, 30515 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/hdc1 1 63 506016 82 Linux swap / Solaris /dev/hdc2 64 30515 244605690 83 Linux
So here are the results. I have run these tests from single-user mode (-b bootflag) so that no other userspace programs would interfere.
Script started on Mon May 22 15:32:29 2006 15:32 (none):~ # mount /dev/hdc2 /D -o barrier 15:32 (none):~ # df -Th / /D Filesystem Type Size Used Avail Use% Mounted on /dev/hda3 xfs 33G 6.7G 27G 21% / /dev/hdc2 xfs 234G 142G 92G 61% /D 15:33 (none):~ # cat /proc/mounts rootfs / rootfs rw 0 0 udev /dev tmpfs rw 0 0 /dev/hda3 / xfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/hdc2 /D xfs rw 0 0 proc /proc proc rw 0 0
A small nitpick BTW, the barrier flag is not listed in /proc/mounts.
hda is always in nobarrier mode. <6>hda: cache flushes not supported <6>hdc: cache flushes supported <5>Filesystem "hda3": Disabling barriers, not supported by the underlying device hdc can do barriers if wanted.
CASE 1: Copying from one disk to another ======================================== Copying a compiled 2.6.17-rc4 tree; 306907 KB in 28566 files in 2090 directories.
15:33 (none):~ # cd /tmp 15:33 (none):/tmp # time rsync -PHSav kernel-source/ /D/kernel-0/ >/dev/null
real 7m31.776s user 0m7.030s sys 0m14.450s
I rebooted here (only here - no more reboots below) to get a cold slate.
15:42 (none):~ # mount /dev/hdc2 /D -o nobarrier 15:42 (none):/tmp # rsync -PHSav kernel-source/ /D/kernel-1/ >/dev/null
real 2m14.401s user 0m7.290s sys 0m14.900s 15:45 (none):/tmp # umount /D
CASE 2: Removing ================ Remove the copies we created in case 1.
15:45 (none):/tmp # mount /dev/hdc2 /D -o barrier 15:45 (none):/tmp # cd /D 15:45 (none):/D # time rm -Rf kernel-0
real 3m31.901s user 0m0.050s sys 0m3.140s 15:49 (none):/D # cd / 15:49 (none):/ # umount /D 15:49 (none):/ # mount /dev/hdc2 /D -o nobarrier 15:49 (none):/ # cd /D 15:49 (none):/D # time rm -Rf kernel-1
real 0m53.471s user 0m0.070s sys 0m1.990s 15:50 (none):/D # cd / 15:50 (none):/ # umount /D
CASE 3: Copying to the same disk ================================ Copy a file from hdc2 to hdc2. The result is _extremely_ interesting. So the 'barrier' thing is only relevant for large sets of files, it seems.
15:50 (none):/ # mount /dev/hdc2 /D -o barrier 15:50 (none):/ # cd /D/Video 15:50 (none):/D/Video # ls -l The* -rw-r--r-- 1 jengelh users 6861772800 Mar 17 23:02 TLK3.iso 15:50 (none):/D/Video # time cp TLK3.iso ../tlk-0.iso
real 8m1.051s user 0m0.670s sys 0m41.120s 15:59 (none):/D/Video # cd / 15:59 (none):/ # umount /D 15:59 (none):/ # mount /dev/hdc2 /D -o nobarrier 15:59 (none):/ # Kcd /D/Video 15:59 (none):/D/Video # time cp TLK3.iso ../tlk-1.iso
real 7m53.275s user 0m0.560s sys 0m40.010s 16:07 (none):/D/Video # cd / 16:08 (none):/ # umount /D 16:08 (none):/ # exit
Script done on Mon May 22 16:08:09 2006
Jan Engelhardt -- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |