lkml.org 
[lkml]   [2010]   [Jun]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: xfs, 2.6.27=>.32 sync write 10 times slowdown [was: xfs, aacraid 2.6.27 => 2.6.32 results in 6 times slowdown]
09.06.2010 11:47, Dave Chinner wrote:
> On Wed, Jun 09, 2010 at 10:43:37AM +0400, Michael Tokarev wrote:
>> 09.06.2010 03:18, Dave Chinner wrote:
>>> On Wed, Jun 09, 2010 at 12:34:00AM +0400, Michael Tokarev wrote:
>> []
>>>> Simple test doing random reads or writes of 4k blocks in a 1Gb
>>>> file located on an xfs filesystem, Mb/sec:
>>>>
>>>> sync direct
>>>> read write write
>>>> 2.6.27 xfs 1.17 3.69 3.80
>>>> 2.6.32 xfs 1.26 0.52 5.10
>>>> ^^^^
>>>> 2.6.32 ext3 1.19 4.91 5.02
>
> Out of curiousity, what does 2.6.34 get on this workload?

2.6.34 works quite well:
2.6.34 xfs 1.14 4.75 5.00

The same is with -o osyncisosync (in .34). Actually,
osyncis[od]sync mount options does not change anything, not
in .32 nor in .34.

> Also, what happens if you test with noop or deadline scheduler,
> rather than cfq (or whichever one you are using)? i.e. is this a
> scheduler regression rather than a filesystem issue?

Using deadline. Switching to noop makes no difference whatsoever.

> Also, a block trace of the sync write workload on both .27 and .32
> would be interesting to see what the difference in IO patterns is...

I see. Will try to collect them. With the limited timeframe I have
to do any testing.

[]
> Well, I normally just create a raid0 lun per disk in those cases,
> hence the luns present the storage to linux as a JBOD....

That's, um, somewhat ugly :)

>> I also experimented with both O_SYNC|O_DIRECT: it is as slow as
>> without O_DIRECT, i.e. O_SYNC makes whole thing slow regardless
>> of other options.
>
> So it's the inode writeback that is causing the slowdown. We've
> recently changed O_SYNC semantics to be real O_SYNC, not O_DSYNC
> as .27 is. I can't remember if that was in 2.6.32 or not, but
> there's definitely a recent change to O_SYNC behaviouri that would
> cause this...

But there are two mount options that seems to control this behavour:
osyncisosync and osyncisdsync. Neither of which - seemingly - makes
no difference.

>> related to block devices or usage of barriers. For XFS it always
>> mounts like this:
>>
>> SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
>> SGI XFS Quota Management subsystem
>> XFS mounting filesystem sda6
>
> So barriers are being issued.

They _are_ being issued, I knew it from the start. What I asked
several times is if there's a way to know if they're _hitting_ the
actual low-level device (disk or raid controller). This is entirely
different story... ;)

>> and for the device in question, it is always like
>>
>> Adaptec aacraid driver 1.1-5[2456]-ms
>> aacraid 0000:03:01.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
>> AAC0: kernel 5.1-0[8832] Feb 1 2006
>
> Old firmware. An update might help.

Well, it worked just fine in .27. So far I see some problem in kernel,
not in controller [firmware]... ;)

Thank you !

/mjt


\
 
 \ /
  Last update: 2010-06-09 21:15    [W:1.151 / U:2.320 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site