[lkml]   [2012]   [Jan]   [2]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    Patch in this message
    Subject[PATCH v5 7/8] mm: Only IPI CPUs to drain local pages if they exist
    Calculate a cpumask of CPUs with per-cpu pages in any zone
    and only send an IPI requesting CPUs to drain these pages
    to the buddy allocator if they actually have pages when
    asked to flush.

    This patch saves 99% of IPIs asking to drain per-cpu
    pages in case of severe memory preassure that leads
    to OOM since in these cases multiple, possibly concurrent,
    allocation requests end up in the direct reclaim code
    path so when the per-cpu pages end up reclaimed on first
    allocation failure for most of the proceeding allocation
    attempts until the memory pressure is off (possibly via
    the OOM killer) there are no per-cpu pages on most CPUs
    (and there can easily be hundreds of them).

    This also has the side effect of shortening the average
    latency of direct reclaim by 1 or more order of magnitude
    since waiting for all the CPUs to ACK the IPI takes a
    long time.

    Tested by running "hackbench 400" on a 4 CPU x86 otherwise
    idle VM and observing the difference between the number
    of direct reclaim attempts that end up in drain_all_pages()
    and those were more then 1/2 of the online CPU had any
    per-cpu page in them, using the vmstat counters introduced
    in the next patch in the series and using proc/interrupts.

    In the test sceanrio, this saved around 500 global IPIs.
    After trigerring an OOM:

    $ cat /proc/vmstat
    pcp_global_drain 627
    pcp_global_ipi_saved 578

    I've also seen the number of drains reach 15k calls
    with the saved percentage reaching 99% when there
    are more tasks running during an OOM kill.

    Signed-off-by: Gilad Ben-Yossef <>
    Acked-by: Christoph Lameter <>
    CC: Chris Metcalf <>
    CC: Peter Zijlstra <>
    CC: Frederic Weisbecker <>
    CC: Russell King <>
    CC: Pekka Enberg <>
    CC: Matt Mackall <>
    CC: Sasha Levin <>
    CC: Rik van Riel <>
    CC: Andi Kleen <>
    CC: Mel Gorman <>
    CC: Andrew Morton <>
    CC: Alexander Viro <>
    CC: Avi Kivity <>
    Christopth Ack was for a previous version that allocated
    the cpumask in drain_all_pages().

    mm/page_alloc.c | 26 +++++++++++++++++++++++++-
    1 files changed, 25 insertions(+), 1 deletions(-)

    diff --git a/mm/page_alloc.c b/mm/page_alloc.c
    index 2b8ba3a..092c331 100644
    --- a/mm/page_alloc.c
    +++ b/mm/page_alloc.c
    @@ -67,6 +67,14 @@ DEFINE_PER_CPU(int, numa_node);

    + * A global cpumask of CPUs with per-cpu pages that gets
    + * recomputed on each drain. We use a global cpumask
    + * for to avoid allocation on direct reclaim code path
    + */
    +static cpumask_var_t cpus_with_pcps;
    * N.B., Do NOT reference the '_numa_mem_' per cpu variable directly.
    @@ -1119,7 +1127,19 @@ void drain_local_pages(void *arg)
    void drain_all_pages(void)
    - on_each_cpu(drain_local_pages, NULL, 1);
    + int cpu;
    + struct per_cpu_pageset *pcp;
    + struct zone *zone;
    + for_each_online_cpu(cpu)
    + for_each_populated_zone(zone) {
    + pcp = per_cpu_ptr(zone->pageset, cpu);
    + if (pcp->pcp.count)
    + cpumask_set_cpu(cpu, cpus_with_pcps);
    + else
    + cpumask_clear_cpu(cpu, cpus_with_pcps);
    + }
    + on_each_cpu_mask(cpus_with_pcps, drain_local_pages, NULL, 1);

    @@ -3623,6 +3643,10 @@ static void setup_zone_pageset(struct zone *zone)
    void __init setup_per_cpu_pageset(void)
    struct zone *zone;
    + int ret;
    + ret = zalloc_cpumask_var(&cpus_with_pcps, GFP_KERNEL);
    + BUG_ON(!ret);


     \ /
      Last update: 2012-01-02 11:29    [W:0.024 / U:228.888 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site