lkml.org 
[lkml]   [2008]   [Mar]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 0/7] IO CPU affinity testing series
    On Wed, Mar 12 2008, Max Krasnyanskiy wrote:
    > Jens Axboe wrote:
    > >Hi,
    > >
    > >Here's a new round of patches to play with io cpu affinity. It can,
    > >as always, also be found in the block git repo. The branch name is
    > >'io-cpu-affinity'.
    > >
    > >The major change since last post is the abandonment of the kthread
    > >approach. It was definitely slower then may 'add IPI to signal remote
    > >block softirq' hack. So I decided to base this on the scalable
    > >smp_call_function_single() that Nick posted. I tweaked it a bit to
    > >make it more suitable for my use and also faster.
    > >
    > >As for functionality, the only change is that I added a bio hint
    > >that the submitter can use to ask for completion on the same CPU
    > >that submitted the IO. Pass in BIO_CPU_AFFINE for that to occur.
    > >
    > >Otherwise the modes are the same as last time:
    > >
    > >- You can set a specific cpumask for queuing IO, and the block layer
    > > will move submitters to one of those CPUs.
    > >- You can set a specific cpumask for completion of IO, in which case
    > > the block layer will move the completion to one of those CPUs.
    > >- You can set rq_affinity mode, in which case IOs will always be
    > > completed on the CPU that submitted them.
    > >
    > >Look in /sys/block/<dev>/queue/ for the three sysfs variables that
    > >modify this behaviour.
    > >
    > >I'd be interested in getting some testing done on this, to see if
    > >it really helps the larger end of the scale. Dave, I know you
    > >have a lot of experience in this area and would appreciate your
    > >input and/or testing. I'm not sure if any of the above modes will
    > >allow you to do what you need for eg XFS - if you want all meta data
    > >IO completed on one (or a set of) CPU(s), then I can add a mode
    > >that will allow you to play with that. Or if something else, give me
    > >some input and we can take it from there!
    >
    > Very cool stuff. I think I can use it for cpu isolation purposes.
    > ie Isolating a cpu from the io activity.
    >
    > You may have noticed that I started a bunch of discussion on CPU isolation.
    > One thing that came out of that is the suggestion to use cpusets for
    > managing this affinity masks. We're still discussing the details, the
    > general idea is to provide extra flags in the cpusets that enable/disable
    > various activities
    > on the cpus that belong to the set.
    >
    > For example in this particular case we'd have something like "cpusets.io"
    > flag that would indicate whether cpus in the set are allowed to to the IO
    > or not.
    > In other words:
    > /dev/cpuset/io (cpus=0,1,2; io=1)
    > /dev/cpuset/no-io (cpus=3,4,5; io=0)
    >
    > I'm not sure whether this makes sense or not. One advantage is that it's
    > more dynamic and more flexible. If for example you add cpu to the io cpuset
    > it will automatically start handling io requests.

    The code posted here works on the queue level, where as you want this to
    be a global setting. So it'll require a bit of extra stuff to handle
    that case, but the base infrastructure would not care.

    > btw What did you mean by "to see if it really helps the larger end of the
    > scale", what problem were you guys trying to solve ? I'm guessing cpu
    > isolation would probably be an unexpected user of io cpu affinity :).

    Nope, I didn't really consider isolation :-)

    It's meant to speed up IO on larger SMP systems by reducing cache line
    contention (or bouncing) by keeping data and/or locks local to a CPU (or
    a set of CPUs).

    --
    Jens Axboe



    \
     
     \ /
      Last update: 2008-03-13 13:17    [W:0.025 / U:180.992 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site