lkml.org 
[lkml]   [2021]   [Dec]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 2/2] sched/fair: Adjust the allowed NUMA imbalance when SD_NUMA spans multiple LLCs
    On Sat, Dec 04, 2021 at 11:40:56AM +0100, Peter Zijlstra wrote:
    > On Wed, Dec 01, 2021 at 03:18:44PM +0000, Mel Gorman wrote:
    > > + /* Calculate allowed NUMA imbalance */
    > > + for_each_cpu(i, cpu_map) {
    > > + int imb_numa_nr = 0;
    > > +
    > > + for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
    > > + struct sched_domain *child = sd->child;
    > > +
    > > + if (!(sd->flags & SD_SHARE_PKG_RESOURCES) && child &&
    > > + (child->flags & SD_SHARE_PKG_RESOURCES)) {
    > > + int nr_groups;
    > > +
    > > + nr_groups = sd->span_weight / child->span_weight;
    > > + imb_numa_nr = max(1U, ((child->span_weight) >> 1) /
    > > + (nr_groups * num_online_nodes()));
    > > + }
    > > +
    > > + sd->imb_numa_nr = imb_numa_nr;
    > > + }
    >
    > OK, so let's see. All domains with SHARE_PKG_RESOURCES set will have
    > imb_numa_nr = 0, all domains above it will have the same value
    > calculated here.
    >
    > So far so good I suppose :-)
    >

    Good start :)

    > Then nr_groups is what it says on the tin; we could've equally well
    > iterated sd->groups and gotten the same number, but this is simpler.
    >

    I also thought it would be clearer.

    > Now, imb_numa_nr is where the magic happens, the way it's written
    > doesn't help, but it's something like:
    >
    > (child->span_weight / 2) / (nr_groups * num_online_nodes())
    >
    > With a minimum value of 1. So the larger the system is, or the smaller
    > the LLCs, the smaller this number gets, right?
    >

    Correct.

    > So my ivb-ep that has 20 cpus in a LLC and 2 nodes, will get: (20 / 2)
    > / (1 * 2) = 10, while the ivb-ex will get: (20/2) / (1*4) = 5.
    >
    > But a Zen box that has only like 4 CPUs per LLC will have 1, regardless
    > of how many nodes it has.
    >

    The minimum of one was to allow a pair of communicating tasks to remain
    on one node even if it's imbalacnced.

    > Now, I'm thinking this assumes (fairly reasonable) that the level above
    > LLC is a node, but I don't think we need to assume this, while also not
    > assuming the balance domain spans the whole machine (yay paritions!).
    >
    > for (top = sd; top->parent; top = top->parent)
    > ;
    >
    > nr_llcs = top->span_weight / child->span_weight;
    > imb_numa_nr = max(1, child->span_weight / nr_llcs);
    >
    > which for my ivb-ep gets me: 20 / (40 / 20) = 10
    > and the Zen system will have: 4 / (huge number) = 1
    >
    > Now, the exp: a / (b / a) is equivalent to a * (a / b) or a^2/b, so we
    > can also write the above as:
    >
    > (child->span_weight * child->span_weight) / top->span_weight;
    >

    Gautham had similar reasoning to calculate the imbalance at each
    higher-level domain instead of using a static value throughout and
    it does make sense. For each level and splitting the imbalance between
    two domains, this works out as


    /*
    * Calculate an allowed NUMA imbalance such that LLCs do not get
    * imbalanced.
    */
    for_each_cpu(i, cpu_map) {
    for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
    struct sched_domain *child = sd->child;

    if (!(sd->flags & SD_SHARE_PKG_RESOURCES) && child &&
    (child->flags & SD_SHARE_PKG_RESOURCES)) {
    struct sched_domain *top = sd;
    unsigned int llc_sq;

    /*
    * nr_llcs = (top->span_weight / llc_weight);
    * imb = (child_weight / nr_llcs) >> 1
    *
    * is equivalent to
    *
    * imb = (llc_weight^2 / top->span_weight) >> 1
    *
    */
    llc_sq = child->span_weight * child->span_weight;
    while (top) {
    top->imb_numa_nr = max(1U,
    (llc_sq / top->span_weight) >> 1);
    top = top->parent;
    }

    break;
    }
    }
    }

    I'll test this and should have results tomorrow.

    --
    Mel Gorman
    SUSE Labs

    \
     
     \ /
      Last update: 2021-12-06 16:40    [W:7.680 / U:0.004 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site