[lkml]   [2012]   [May]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: Plumbers: Tweaking scheduler policy micro-conf RFP
    On Wed, May 16, 2012 at 11:30 AM, Peter Zijlstra <> wrote:
    > The reason I want to push this into the arch is that the current one
    > size fits all topology really doesn't fit.


    Ingo, please don't take any of these patches if they are starting to
    make NUMA scheduling be some arch-specific crap.

    Peter - you're way off base. You are totally and utterly wrong for
    several reasons:

    - x86 isn't that special. In fact, x86 covers almost all the possible
    NUMA architectures just within x86, so making some arch-specific NUMA
    crap is *idiotic*.

    Your argument that SMT, multi-core with NUMA effects within the
    core, yadda yadda is somehow x86-specific is pure crap. Everybody else
    already does that or will do that. More importantly, most of the other
    architectures don't even *matter* enough for them to ever write their
    own NUMA scheduler.

    - NUMA topology details aren't even that important!

    Christ, people. You're never going to be perfect anyway. And
    outside of some benchmarks, and some trivial loads, the loads aren't
    going to be well-behaved to really let you even try.

    You don't even know what the right scheduling policy should be. We
    already know that even the *existing* power-aware scheduling is pure
    crap and nobody believes it actually works.

    Trying to redesign something from scratch when you don't even
    understand it, AND THEN MAKING IT ARCH-SPECIFIC, is just f*cking
    moronic. The *only* thing that will ever result in is some crap code
    that handles the one or two machines you tested it on right, for the
    one or two loads you tested it with.

    If you cannot make it simple enough, and generic enough, that it works
    reasonably well for POWER and s390 (and ARM), don't even bother.


    If you hate the s390 'book' domain and it really causes problems, rip
    it out. NUMA code has *never* worked well. And I'm not talking just
    Linux. I'm talking about all the other systems that tried to to do it,
    and tried too effin hard.

    Stop the idiocy. Make things *simpler* and *less* capable instead of
    trying to cover some "real" topology. Nobody cares in real life.
    Seriously. And the hardware people are going to keep on making things
    different. Don't try to build up some perfect NUMA topology and then
    try to see how insanely well you can match a particular machine. Make
    some generic "roughly like this" topology with (say) four three of
    NUMAness, and then have architectures say "this is roughly what my
    machine looks like".

    And if you cannot do that kind of "roughly generic NUMA", then stop
    working on this crap immediately. Rather than waste everybodys time
    with some crazy arch-specific random scheduling.

    Make the levels be something like

    (a) "share core resources" (SMT or shared inner cache, like a shared
    L2 when there is a big L3)
    (b) share socket
    (c) shared board/node

    and don't try to make it any more clever. Don't try to describe just
    *how* much resources are shared. Nobody cares at that level. If it's
    not a difference of an order of magnitude, it's damn well the same
    f*cking thing! Don't think that SMT is all that different from "two
    cores sharing the same L2". Sure, they are different, but you won't
    find real-life loads that care enough for us to ever bother with the

    Don't try to describe it any more than that. Seriously. Any time you
    add "implementation detail" level knowledge (like the s390 'book' or
    the various forms of SMT vs "shared decorer" (Bulldozer/Trinity) vs
    "true cores sharing L2" (clusters of cores sharing an L2, with a big
    shared L3 within the socket), you're just shooting yourself in the
    foot. You're adding crap that shouldn't be added.

    In particular, don't even try to give random "weights" to how close
    things are to each other. Sure, you can parse (and generate) those
    complex NUMA tables, but nobody is *ever* smart enough to really use
    them. Once you move data between boards/nodes, screw the number of
    hops. You are NOT going to get some scheduling decision right that
    says "node X is closer to node Y than to node Z". Especially since the
    load is invariably going to access non-node memory too *anyway*.

    Seriously, if you think you need some complex data structures to
    describe the relationship between cores, you're barking up the wrong

    Make it a simple three-level tree like the above. No weights. No
    *nothing*. If there isn't an order of magnitude difference in
    performance and/or a *major* power domain issue, they're at the same
    level. Nothing smarter than that, because it will just be f*cking
    stupid, and it will be a nightmare to maintain, and nobody will
    understand it anyway.

    And the *only* thing that should be architecture-specific is the
    architecture CPU init code that says "ok, this core is a SMT thread,
    so it is in the same (a) level NUMA domain as that other core".

    I'm very very serious about this. Try to make the scheduler have a
    *simple* model that people can actually understand. For example, maybe
    it can literally be a multi-level balancing thing, where the per-cpu
    runqueues are grouped into a "shared core resources" balancer that
    balances within the SMT or shared-L2 domain. And then there's an
    upper-level balancer (that runs much more seldom) that is written to
    balances within the socket. And then one that balances within the
    node/board. And finally one that balances across boards.

    Then, at each level, just say "spread out" (unrelated loads that want
    maximum throughput) vs "try to group together" (for power reasons and
    to avoid cache bouncing for related loads).

    I dunno what the details need to be, and the above is just some random
    off-the-cuff example.

    But what I *do* know that we don't want any arch-specific code. And I
    *do* know that the real world simply isn't simple enough that we could
    ever do a perfect job, so don't even try - instead aim for
    "understandable, maintainable, and gets the main issues roughly


     \ /
      Last update: 2012-05-19 19:21    [W:0.029 / U:10.636 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site