[lkml]   [2011]   [Jan]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [RFC] mm: Make vm_acct_memory scalable for large memory allocations
    On Wed, 26 Jan 2011 14:51:59 -0800
    Tim Chen <> wrote:

    > During testing of concurrent malloc/free by multiple processes on a 8
    > socket NHM-EX machine (8cores/socket, 64 cores total), I noticed that
    > malloc of large memory (e.g. 32MB) did not scale well. A test patch
    > included here increased 32MB mallocs/free with 64 concurrent processes
    > from 69K operations/sec to 4066K operations/sec on 2.6.37 kernel, and
    > eliminated the cpu cycles contending for spin_lock in the vm_commited_as
    > percpu_counter.

    This seems like a pretty dumb test case. We have 64 cores sitting in a
    loop "allocating" 32MB of memory, not actually using that memory and
    then freeing it up again.

    Any not-completely-insane application would actually _use_ the memory.
    Which involves pagefaults, page allocations and much memory traffic
    modifying the page contents.

    Do we actually care?

    > Spin lock contention occurs when vm_acct_memory increments/decrements
    > the percpu_counter vm_committed_as by the number of pages being
    > used/freed. Theoretically vm_committed_as is a percpu_counter and should
    > streamline the concurrent update by using the local counter in
    > vm_commited_as. However, if the update is greater than
    > percpu_counter_batch limit, then it will overflow into the global count
    > in vm_commited_as. Currently percpu_counter_batch is non-configurable
    > and hardcoded to 2*num_online_cpus. So any update of vm_commited_as by
    > more than 256 pages will cause overflow in my test scenario which has
    > 128 logical cpus.
    > In the patch, I have set an enlargement multiplication factor for
    > vm_commited_as's batch limit. I limit the sum of all local counters up
    > to 5% of the total pages before overflowing into the global counter.
    > This will avoid the frequent contention of the spin_lock in
    > vm_commited_as. Some additional work will need to be done to make
    > setting of this multiplication factor cpu hotplug aware. Advise on
    > better approaches are welcomed.
    > ...
    > Signed-off-by: Tim Chen <>
    > diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h
    > index 46f6ba5..5a892d8 100644
    > --- a/include/linux/percpu_counter.h
    > +++ b/include/linux/percpu_counter.h
    > @@ -21,6 +21,7 @@ struct percpu_counter {
    > struct list_head list; /* All percpu_counters are on a list */
    > #endif
    > + u32 multibatch;
    > s32 __percpu *counters;
    > };

    I dunno. Wouldn't it be better to put a `batch' field into
    percpu_counter and then make the global percpu_counter_batch just go

    That would require modifying each counter's `batch' at cpuhotplug time,
    while somehow retaining the counter's user's intent. So perhaps the
    counter would need two fields - original_batch and operating_batch or

     \ /
      Last update: 2011-01-28 00:39    [W:0.025 / U:0.096 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site