Messages in this thread |  | | | Date | Thu, 13 Mar 2008 13:13:21 +0100 | | From | Jens Axboe <> | | Subject | Re: [PATCH 0/7] IO CPU affinity testing series |
| |
On Wed, Mar 12 2008, Max Krasnyanskiy wrote: > Jens Axboe wrote: > >Hi, > > > >Here's a new round of patches to play with io cpu affinity. It can, > >as always, also be found in the block git repo. The branch name is > >'io-cpu-affinity'. > > > >The major change since last post is the abandonment of the kthread > >approach. It was definitely slower then may 'add IPI to signal remote > >block softirq' hack. So I decided to base this on the scalable > >smp_call_function_single() that Nick posted. I tweaked it a bit to > >make it more suitable for my use and also faster. > > > >As for functionality, the only change is that I added a bio hint > >that the submitter can use to ask for completion on the same CPU > >that submitted the IO. Pass in BIO_CPU_AFFINE for that to occur. > > > >Otherwise the modes are the same as last time: > > > >- You can set a specific cpumask for queuing IO, and the block layer > > will move submitters to one of those CPUs. > >- You can set a specific cpumask for completion of IO, in which case > > the block layer will move the completion to one of those CPUs. > >- You can set rq_affinity mode, in which case IOs will always be > > completed on the CPU that submitted them. > > > >Look in /sys/block/<dev>/queue/ for the three sysfs variables that > >modify this behaviour. > > > >I'd be interested in getting some testing done on this, to see if > >it really helps the larger end of the scale. Dave, I know you > >have a lot of experience in this area and would appreciate your > >input and/or testing. I'm not sure if any of the above modes will > >allow you to do what you need for eg XFS - if you want all meta data > >IO completed on one (or a set of) CPU(s), then I can add a mode > >that will allow you to play with that. Or if something else, give me > >some input and we can take it from there! > > Very cool stuff. I think I can use it for cpu isolation purposes. > ie Isolating a cpu from the io activity. > > You may have noticed that I started a bunch of discussion on CPU isolation. > One thing that came out of that is the suggestion to use cpusets for > managing this affinity masks. We're still discussing the details, the > general idea is to provide extra flags in the cpusets that enable/disable > various activities > on the cpus that belong to the set. > > For example in this particular case we'd have something like "cpusets.io" > flag that would indicate whether cpus in the set are allowed to to the IO > or not. > In other words: > /dev/cpuset/io (cpus=0,1,2; io=1) > /dev/cpuset/no-io (cpus=3,4,5; io=0) > > I'm not sure whether this makes sense or not. One advantage is that it's > more dynamic and more flexible. If for example you add cpu to the io cpuset > it will automatically start handling io requests.
The code posted here works on the queue level, where as you want this to be a global setting. So it'll require a bit of extra stuff to handle that case, but the base infrastructure would not care.
> btw What did you mean by "to see if it really helps the larger end of the > scale", what problem were you guys trying to solve ? I'm guessing cpu > isolation would probably be an unexpected user of io cpu affinity :).
Nope, I didn't really consider isolation :-)
It's meant to speed up IO on larger SMP systems by reducing cache line contention (or bouncing) by keeping data and/or locks local to a CPU (or a set of CPUs).
-- Jens Axboe
|  |