lkml.org 
[lkml]   [2008]   [Mar]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH 0/7] IO CPU affinity testing series
On Wed, Mar 12 2008, Max Krasnyanskiy wrote:
> Jens Axboe wrote:
> >Hi,
> >
> >Here's a new round of patches to play with io cpu affinity. It can,
> >as always, also be found in the block git repo. The branch name is
> >'io-cpu-affinity'.
> >
> >The major change since last post is the abandonment of the kthread
> >approach. It was definitely slower then may 'add IPI to signal remote
> >block softirq' hack. So I decided to base this on the scalable
> >smp_call_function_single() that Nick posted. I tweaked it a bit to
> >make it more suitable for my use and also faster.
> >
> >As for functionality, the only change is that I added a bio hint
> >that the submitter can use to ask for completion on the same CPU
> >that submitted the IO. Pass in BIO_CPU_AFFINE for that to occur.
> >
> >Otherwise the modes are the same as last time:
> >
> >- You can set a specific cpumask for queuing IO, and the block layer
> > will move submitters to one of those CPUs.
> >- You can set a specific cpumask for completion of IO, in which case
> > the block layer will move the completion to one of those CPUs.
> >- You can set rq_affinity mode, in which case IOs will always be
> > completed on the CPU that submitted them.
> >
> >Look in /sys/block/<dev>/queue/ for the three sysfs variables that
> >modify this behaviour.
> >
> >I'd be interested in getting some testing done on this, to see if
> >it really helps the larger end of the scale. Dave, I know you
> >have a lot of experience in this area and would appreciate your
> >input and/or testing. I'm not sure if any of the above modes will
> >allow you to do what you need for eg XFS - if you want all meta data
> >IO completed on one (or a set of) CPU(s), then I can add a mode
> >that will allow you to play with that. Or if something else, give me
> >some input and we can take it from there!
>
> Very cool stuff. I think I can use it for cpu isolation purposes.
> ie Isolating a cpu from the io activity.
>
> You may have noticed that I started a bunch of discussion on CPU isolation.
> One thing that came out of that is the suggestion to use cpusets for
> managing this affinity masks. We're still discussing the details, the
> general idea is to provide extra flags in the cpusets that enable/disable
> various activities
> on the cpus that belong to the set.
>
> For example in this particular case we'd have something like "cpusets.io"
> flag that would indicate whether cpus in the set are allowed to to the IO
> or not.
> In other words:
> /dev/cpuset/io (cpus=0,1,2; io=1)
> /dev/cpuset/no-io (cpus=3,4,5; io=0)
>
> I'm not sure whether this makes sense or not. One advantage is that it's
> more dynamic and more flexible. If for example you add cpu to the io cpuset
> it will automatically start handling io requests.

The code posted here works on the queue level, where as you want this to
be a global setting. So it'll require a bit of extra stuff to handle
that case, but the base infrastructure would not care.

> btw What did you mean by "to see if it really helps the larger end of the
> scale", what problem were you guys trying to solve ? I'm guessing cpu
> isolation would probably be an unexpected user of io cpu affinity :).

Nope, I didn't really consider isolation :-)

It's meant to speed up IO on larger SMP systems by reducing cache line
contention (or bouncing) by keeping data and/or locks local to a CPU (or
a set of CPUs).

--
Jens Axboe



\
 
 \ /
  Last update: 2008-03-13 13:17    [from the cache]
©2003-2011 Jasper Spaans. Advertise on this site