lkml.org 
[lkml]   [2009]   [Nov]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH] make crypto unplug fix V3
On Fri, Nov 13, 2009 at 01:40:04PM -0500, Chris Mason wrote:
> On Fri, Nov 13, 2009 at 05:34:46PM +0000, Mel Gorman wrote:
> > On Fri, Nov 13, 2009 at 07:58:12AM -0500, Chris Mason wrote:
> > > This is still likely to set your dm data on fire. It is only meant for
> > > testers that start with mkfs and don't have any valuable dm data.
> > >
> >
> > The good news is that my room remains fire-free. Despite swap also
> > running from dm-crypt, I had no corruption or instability issues.
>
> Ok, definitely not so convincing I'd try and shove it into a late rc.
>
> >
> > Here is an updated set of results for fake-gitk running.
> >
> > X86
> > 2.6.30-0000000-force-highorder Elapsed:12:08.908 Failures:0
> > 2.6.31-0000000-force-highorder Elapsed:10:56.283 Failures:0
> > 2.6.31-0000006-dm-crypt-unplug Elapsed:11:51.653 Failures:0
> > 2.6.31-0000012-pgalloc-2.6.30 Elapsed:12:26.587 Failures:0
> > 2.6.31-0000123-congestion-both Elapsed:10:55.298 Failures:0
> > 2.6.31-0001234-kswapd-quick-recheck Elapsed:18:01.523 Failures:0
> > 2.6.31-0123456-dm-crypt-unplug Elapsed:10:45.720 Failures:0
> > 2.6.31-revert-8aa7e847 Elapsed:15:08.020 Failures:0
> > 2.6.32-rc6-0000000-force-highorder Elapsed:16:20.765 Failures:4
> > 2.6.32-rc6-0000006-dm-crypt-unplug Elapsed:13:42.920 Failures:0
> > 2.6.32-rc6-0000012-pgalloc-2.6.30 Elapsed:16:13.380 Failures:1
> > 2.6.32-rc6-0000123-congestion-both Elapsed:18:39.118 Failures:0
> > 2.6.32-rc6-0001234-kswapd-quick-recheck Elapsed:15:04.398 Failures:0
> > 2.6.32-rc6-0123456-dm-crypt-unplug Elapsed:12:50.438 Failures:0
> > 2.6.32-rc6-revert-8aa7e847 Elapsed:20:50.888 Failures:0
> >
> > X86-64
> > 2.6.30-0000000-force-highorder Elapsed:10:37.300 Failures:0
> > 2.6.31-0000000-force-highorder Elapsed:08:49.338 Failures:0
> > 2.6.31-0000006-dm-crypt-unplug Elapsed:09:37.840 Failures:0
> > 2.6.31-0000012-pgalloc-2.6.30 Elapsed:15:49.690 Failures:0
> > 2.6.31-0000123-congestion-both Elapsed:09:18.790 Failures:0
> > 2.6.31-0001234-kswapd-quick-recheck Elapsed:08:39.268 Failures:0
> > 2.6.31-0123456-dm-crypt-unplug Elapsed:08:20.965 Failures:0
> > 2.6.31-revert-8aa7e847 Elapsed:08:07.457 Failures:0
> > 2.6.32-rc6-0000000-force-highorder Elapsed:18:29.103 Failures:1
> > 2.6.32-rc6-0000006-dm-crypt-unplug Elapsed:25:53.515 Failures:3
> > 2.6.32-rc6-0000012-pgalloc-2.6.30 Elapsed:19:55.570 Failures:6
> > 2.6.32-rc6-0000123-congestion-both Elapsed:17:29.255 Failures:2
> > 2.6.32-rc6-0001234-kswapd-quick-recheck Elapsed:14:41.068 Failures:0
> > 2.6.32-rc6-0123456-dm-crypt-unplug Elapsed:15:48.028 Failures:1
> > 2.6.32-rc6-revert-8aa7e847 Elapsed:14:48.647 Failures:0
> >
> > The numbering in the kernel indicates what patches are applied. I tested
> > the dm-crypt patch both in isolation and in combination with the patches
> > in this series.
> >
> > Basically, the dm-crypt-unplug makes a small difference in performance
> > overall, mostly slight gains and losses. There was one massive regression
> > with the dm-crypt patch applied to 2.6.32-rc6 but at the moment, I don't
> > know what that is.
>
> How consistent are your numbers between runs? I was trying to match
> this up with your last email and things were pretty different.
>

The figures from the first mail were based on kernels that were not
instrumented. It so happened that this run was based on an instrumented
kernel to get the congestion_wait figures so the results are different.

However, the results vary a lot. In some cases, it will spike just as
2.6.32-rc6-0000006-dm-crypt-unplug did. The reported figure is the
average of 4 fake-gitk runs. I don't have the standard deviation handy
but it's high.

> >
> > In general, the patch reduces the amount of time direct reclaimers are
> > spending on congestion_wait.
> >
> > > It includes my patch from last night, along with changes to force dm to
> > > unplug when its IO queues empty.
> > >
> > > The problem goes like this:
> > >
> > > Process: submit read bio
> > > dm: put bio onto work queue
> > > process: unplug
> > > dm: work queue finds bio, does a generic_make_request
> > >
> > > The end result is that we miss the unplug completely. dm-crypt needs to
> > > unplug for sync bios. This patch also changes it to unplug whenever the
> > > queue is empty, which is far from ideal but better than missing the
> > > unplugs.
> > >
> > > This doesn't completely fix io stalls I'm seeing with dm-crypt, but its
> > > my best guess. If it works, I'll break it up and submit for real to
> > > the dm people.
> > >
> >
> > Out of curiousity, how are you measuring IO stalls? In the tests I'm doing,
> > the worker processes output their progress and it should be at a steady
> > rate. I considered a stall to be an excessive delay between updates which
> > is a pretty indirect measure.
>
> I just setup a crypto disk and did dd if=/dev/zero of=/mnt/foo bs=1M
>
> If you watch vmstat 1, there's supposed to be a constant steam of IO to
> the disk. If a whole second goes by with zero IO, we're doing something
> wrong, I get a number of multi-second stalls where we are just waiting
> for IO to happen.
>

Ok, cool.

> Most of the time I was able to catch a sysrq-w for it, someone was
> waiting on a read to finish. It isn't completely clear to me if the
> unplugging is working properly.
>

The machines I'm using are now tied up doing the same tests with dm-crypt
and then I need to rerun with dm-crypt with the changed patches. It'll be
early next week before the machines free up again for me to investigate your
patch more but initial results look promising at least.

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab


\
 
 \ /
  Last update: 2009-11-13 21:31    [W:1.003 / U:0.328 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site