Messages in this thread | | | Date | Wed, 15 Apr 2009 08:26:49 +0200 | From | Jens Axboe <> | Subject | Re: tiobench read 50% regression with 2.6.30-rc1 |
| |
On Wed, Apr 15 2009, Jeff Moyer wrote: > Jens Axboe <jens.axboe@oracle.com> writes: > > > On Fri, Apr 10 2009, Zhang, Yanmin wrote: > >> On Thu, 2009-04-09 at 11:57 +0200, Jens Axboe wrote: > >> > On Thu, Apr 09 2009, Zhang, Yanmin wrote: > >> > > Comparing with 2.6.29's result, tiobench (read) has about 50% regression > >> > > with 2.6.30-rc1 on all my machines. Bisect down to below patch. > >> > > > >> > > b029195dda0129b427c6e579a3bb3ae752da3a93 is first bad commit > >> > > commit b029195dda0129b427c6e579a3bb3ae752da3a93 > >> > > Author: Jens Axboe <jens.axboe@oracle.com> > >> > > Date: Tue Apr 7 11:38:31 2009 +0200 > >> > > > >> > > cfq-iosched: don't let idling interfere with plugging > >> > > > >> > > When CFQ is waiting for a new request from a process, currently it'll > >> > > immediately restart queuing when it sees such a request. This doesn't > >> > > work very well with streamed IO, since we then end up splitting IO > >> > > that would otherwise have been merged nicely. For a simple dd test, > >> > > this causes 10x as many requests to be issued as we should have. > >> > > Normally this goes unnoticed due to the low overhead of requests > >> > > at the device side, but some hardware is very sensitive to request > >> > > sizes and there it can cause big slow downs. > >> > > > >> > > > >> > > > >> > > Command to start the testing: > >> > > #tiotest -k0 -k1 -k3 -f 80 -t 32 > >> > > > >> > > It's a multi-threaded program and starts 32 threads. Every thread does I/O > >> > > on its own 80MB file. > >> The files should be created before the testing and pls. drop page caches > >> by "echo 3 >/proc/sys/vm/drop_caches" before testing. > >> > >> > > >> > It's not a huge surprise that we regressed there. I'll get this fixed up > >> > next week. Can you I talk you into trying to change the 'quantum' sysfs > >> > variable for the drive? It's in /sys/block/xxx/queue/iosched where xxx > >> > is your drive(s). It's set to 4, if you could try progressively larger > >> > settings and retest, that would help get things started. > >> I tried 4,8,16,64,128 and didn't find result difference. > > > > Can you try with this patch? > > > > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c > > index a4809de..66f00e5 100644 > > --- a/block/cfq-iosched.c > > +++ b/block/cfq-iosched.c > > @@ -1905,10 +1905,17 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq, > > * Remember that we saw a request from this process, but > > * don't start queuing just yet. Otherwise we risk seeing lots > > * of tiny requests, because we disrupt the normal plugging > > - * and merging. > > + * and merging. If the request is already larger than a single > > + * page, let it rip immediately. For that case we assume that > > + * merging is already done. > > */ > > - if (cfq_cfqq_wait_request(cfqq)) > > + if (cfq_cfqq_wait_request(cfqq)) { > > + if (blk_rq_bytes(rq) > PAGE_CACHE_SIZE) { > > + del_timer(&cfqd->idle_slice_timer); > > + blk_start_queueing(cfqd->queue); > > + } > > cfq_mark_cfqq_must_dispatch(cfqq); > > + } > > } else if (cfq_should_preempt(cfqd, cfqq, rq)) { > > /* > > * not the active queue - expire current slice if it is > > I tested this using iozone to read a file from an NFS client. The > iozone command line was: > iozone -s 2000000 -r 64 -f /mnt/test/testfile -i 1 -w > > The numbers in the nfsd's row represent the number of nfsd threads. I > included numbers for the deadline scheduler as well for comparison. > > v2.6.29 > > nfsd's | 1 | 2 | 4 | 8 > --------+---------------+-------+------ > cfq | 91356 | 66391 | 61942 | 51674 > deadline| 43207 | 67436 | 96289 | 107784 > > 2.6.30-rc1 > > nfsd's | 1 | 2 | 4 | 8 > --------+---------------+-------+------ > cfq | 43127 | 22354 | 20858 | 21179 > deadline| 43732 | 68059 | 76659 | 83231 > > 2.6.30-rc1 + cfq fix > > nfsd's | 1 | 2 | 4 | 8 > --------+-----------------+-------+------ > cfq | 114602 | 102280 | 43479 | 43160 > > As you can see, for 1 and 2 threads, the patch *really* helps out. We > still don't get back the performance for 4 and 8 nfsd threads, though. > It's interesting to note that the deadline scheduler regresses for 4 and > 8 threads, as well. I think we've still got some digging to do.
Wow, that does indeed look pretty good!
> I'll try the cfq close cooperator patches next.
I have a pending update on the coop patch that isn't pushed out yet, I hope to have it finalized and tested later today. Hopefully, with that, we should be able to maintain > 100Mb/sec for 4 and 8 threads.
-- Jens Axboe
| |