Messages in this thread | | | Date | Wed, 09 Jun 2010 10:43:37 +0400 | From | Michael Tokarev <> | Subject | Re: xfs, 2.6.27=>.32 sync write 10 times slowdown [was: xfs, aacraid 2.6.27 => 2.6.32 results in 6 times slowdown] |
| |
09.06.2010 03:18, Dave Chinner wrote: > On Wed, Jun 09, 2010 at 12:34:00AM +0400, Michael Tokarev wrote: [] >> Simple test doing random reads or writes of 4k blocks in a 1Gb >> file located on an xfs filesystem, Mb/sec: >> >> sync direct >> read write write >> 2.6.27 xfs 1.17 3.69 3.80 >> 2.6.32 xfs 1.26 0.52 5.10 >> ^^^^ >> 2.6.32 ext3 1.19 4.91 5.02 >> >> Note the 10 times difference between O_SYNC and O_DIRECT writes >> in 2.6.32. This is, well, huge difference, and this is where >> the original slowdown comes from, apparently. > > Are you running on the raw block device, or on top of LVM/DM/MD to > split up the space on the RAID drive? DM+MD have grown barrier > support since 2.6.27, so it may be that barriers are now being > passed down to the raid hardware on 2.6.32 and they never were on > 2.6.27. Can you paste the output of dmesg when the XFS filesystem in
That's why I asked how to tell if barriers are actually hitting the device in question.
No, this is the only machine where DM/MD is _not_ used. On all other machines we use MD software raid, this machine comes with an onboard raid controller that does not work in JBOD mode so I weren't able to use linux software raid. This is XFS on top of Adaptec RAID card, nothing in-between.
Also, as I mentioned in the previous email, remounting with nobarrier makes no difference whatsoever.
(Another side note here - I discovered that unknown options are silently ignored in "remount mode" while correctly rejected in "plain mount" mode, -- it looks like a kernel bug actually, but it's entirely different issue).
> question is mounted on both 2.6.27 and 2.6.32 so we can see if > there is a difference in the use of barriers? > > Also, remember that O_DIRECT does not imply O_SYNC. O_DIRECT writes > only write data, while O_SYNC will also write metadata and/or the > log.
I know this. I also found osyncisosync and osyncisdsync mount options, and when I try to use the latter, kernel tells it's the default and hence deprecated. I don't need metadata updates, but it _looks_ like the system is doing such updates (with barriers or flushes?) anyway even when mounted with -o osyncisdsync it behaves the same: very slow.
I also experimented with both O_SYNC|O_DIRECT: it is as slow as without O_DIRECT, i.e. O_SYNC makes whole thing slow regardless of other options.
I looked at the dmesg outputs, and there's no relevant differences related to block devices or usage of barriers. For XFS it always mounts like this:
SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled SGI XFS Quota Management subsystem XFS mounting filesystem sda6
and for the device in question, it is always like
Adaptec aacraid driver 1.1-5[2456]-ms aacraid 0000:03:01.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24 AAC0: kernel 5.1-0[8832] Feb 1 2006 AAC0: monitor 5.1-0[8832] AAC0: bios 5.1-0[8832] AAC0: serial 267BE0 AAC0: Non-DASD support enabled. AAC0: 64bit support enabled. AAC0: 64 Bit DAC enabled scsi0 : aacraid scsi 0:0:0:0: Direct-Access Adaptec f0500 V1.0 PQ: 0 ANSI: 2 sd 0:0:0:0: [sda] 286715904 512-byte hardware sectors (146799 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA sda: sda1 sda2 sda3 < sda5 sda6 >
There are tons of other differences, but that is to be expected (like format of CPU topology printing which is changed between .27 and .32).
Thanks!
/mjt
| |