lkml.org 
[lkml]   [2011]   [Jan]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: [performance bug] kernel building regression on 64 LCPUs machine
From
On Fri, Jan 21, 2011 at 8:52 AM, Alex,Shi <alex.shi@intel.com> wrote:
> Sorry for forgetting Jan and tytso. added again.
>
> On Fri, 2011-01-21 at 15:47 +0800, Alex,Shi wrote:
>> On Fri, 2011-01-21 at 15:23 +0800, Corrado Zoccolo wrote:
>> > On Wed, Jan 19, 2011 at 2:55 AM, Alex,Shi <alex.shi@intel.com> wrote:
>> > > Shaohua and I tested kernel building performance on latest kernel. and
>> > > found it is drop about 15% on our 64 LCPUs NHM-EX machine on ext4 file
>> > > system. We find this performance dropping is due to commit
>> > > 749ef9f8423054e326f. If we revert this patch or just change the
>> > > WRITE_SYNC back to WRITE in jbd2/commit.c file. the performance can be
>> > > recovered.
>> > >
>> > > iostat report show with the commit, read request merge number increased
>> > > and write request merge dropped. The total request size increased and
>> > > queue length dropped. So we tested another patch: only change WRITE_SYNC
>> > > to WRITE_SYNC_PLUG in jbd2/commit.c, but nothing effected.
>> > >
>> > > we didn't test deadline IO mode, just test cfq. seems insert write
>> > > request into sync queue effect much read performance, but we don't know
>> > > details. What's your comments of this?
>> > >
>> > > iostat of .37 kernel:
>> > > rrqm/s   wrqm/s r/s     w/s     rMB/s  wMB/s avgrq-sz avgqu-sz   await  svctm  %util
>> > > 22.5772 96.46 92.3742 14.747  1.0048  0.439474 34.8557 0.18078 3.8076 0.30447 2.94302
>> > > iostat of commit reverted .37:
>> > > 26.6223 80.21 107.875 6.03538 1.51415 0 41.3275 0.153385 3.80569 0.377231 3.22323
>> >
>> > From these numbers, it seems to me that with the patch reverted, the
>> > write bandwidth is really low, and probably you are keeping most
>> > written files in the buffer cache during the whole compile, while the
>> > non-reverted kernel is making progress in writing out the files. So
>> > the 'improved' read bandwidth is due to unfairness w.r.t. writes.
>> > Does the result change if you add a final sync and time that as well,
>> > in order to see the full time to make it on disk?
>>
>> Agree with your guess, but kbuild is such kind of benchmark, we can not
>> change its behavior. :(
Why not? Benchmarks should try to model real workloads, and after I
build a kernel, I run lilo, that performs a sync (and typically takes
several seconds). If I used grub, I would pay the sync while
unmounting the filesystem, but I would experience the multi-second
delay anyway.

>>
>>
>> >
>> > I think that in a more extreme test where you end up filling all the
>> > buffer cache with written data, you will see much longer stalls with
>> > the revert than without.
>>
>> Have to do this? and if so, it is not kbuild. :)
It might be, on a machine with fewer ram. Can you try running with
just 1G of ram?
>>
>> BTW, the Jan's patch has a little improvement on kbuild. In many time
>> testing, it seems about 3% improving.
The fact is, does it improve (or at least is a noop) also on
constrained memory systems?

>> The average iostat output of Jan's patch:
>>  rrqm/s   wrqm/s r/s     w/s     rMB/s  wMB/s avgrq-sz avgqu-sz   await  svctm  %util
>> 23.419726 87.450685 96.164521 6.748493 1.046438 0.370137 45.182192 0.200685 6.848767 0.394110 3.072192
>>
>> >
>> > Thanks,
>> > Corrado
>> >
>> > >
>> > > vmstat report show, read bandwidth dropping:
>> > > vmstat of .37:
>> > >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>> > > 3.4 52.6 0.0 64303937.0 16466.7 121544.5 0.0 0.0 2102.7 1914.6 7414.1 3185.7 2.0 1.0 80.3 16.7 0.0
>> > > vmstat of revert all from .37
>> > > 2.2 35.8 0.0 64306767.4 17265.6 126101.2 0.0 0.0 2415.8 1619.1 8532.2 3556.2 2.5 1.1 83.0 13.3 0.0
>> > >
>> > > Regards
>> > > Alex
>> > >
>> > > ===
>> > > diff --git a/fs/jbd/commit.c b/fs/jbd/commit.c
>> > > index 34a4861..27ac2f3 100644
>> > > --- a/fs/jbd/commit.c
>> > > +++ b/fs/jbd/commit.c
>> > > @@ -294,7 +294,7 @@ void journal_commit_transaction(journal_t *journal)
>> > >        int first_tag = 0;
>> > >        int tag_flag;
>> > >        int i;
>> > > -       int write_op = WRITE_SYNC;
>> > > +       int write_op = WRITE;
>> > >
>> > >        /*
>> > >         * First job: lock down the current transaction and wait for
>> > > diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
>> > > index f3ad159..69ff08e 100644
>> > > --- a/fs/jbd2/commit.c
>> > > +++ b/fs/jbd2/commit.c
>> > > @@ -329,7 +329,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
>> > >        int tag_bytes = journal_tag_bytes(journal);
>> > >        struct buffer_head *cbh = NULL; /* For transactional checksums */
>> > >        __u32 crc32_sum = ~0;
>> > > -       int write_op = WRITE_SYNC;
>> > > +       int write_op = WRITE;
>> > >
>> > >        /*
>> > >         * First job: lock down the current transaction and wait for
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>
>
>



--
__________________________________________________________________________

dott. Corrado Zoccolo                          mailto:czoccolo@gmail.com
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------
The self-confidence of a warrior is not the self-confidence of the average
man. The average man seeks certainty in the eyes of the onlooker and calls
that self-confidence. The warrior seeks impeccability in his own eyes and
calls that humbleness.
                               Tales of Power - C. Castaneda
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2011-01-21 09:17    [W:1.747 / U:0.192 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site