[lkml]   [2011]   [Jan]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: [RFC] mm: Make vm_acct_memory scalable for large memory allocations
On Wed, 26 Jan 2011 14:51:59 -0800
Tim Chen <> wrote:

> During testing of concurrent malloc/free by multiple processes on a 8
> socket NHM-EX machine (8cores/socket, 64 cores total), I noticed that
> malloc of large memory (e.g. 32MB) did not scale well. A test patch
> included here increased 32MB mallocs/free with 64 concurrent processes
> from 69K operations/sec to 4066K operations/sec on 2.6.37 kernel, and
> eliminated the cpu cycles contending for spin_lock in the vm_commited_as
> percpu_counter.

This seems like a pretty dumb test case. We have 64 cores sitting in a
loop "allocating" 32MB of memory, not actually using that memory and
then freeing it up again.

Any not-completely-insane application would actually _use_ the memory.
Which involves pagefaults, page allocations and much memory traffic
modifying the page contents.

Do we actually care?

> Spin lock contention occurs when vm_acct_memory increments/decrements
> the percpu_counter vm_committed_as by the number of pages being
> used/freed. Theoretically vm_committed_as is a percpu_counter and should
> streamline the concurrent update by using the local counter in
> vm_commited_as. However, if the update is greater than
> percpu_counter_batch limit, then it will overflow into the global count
> in vm_commited_as. Currently percpu_counter_batch is non-configurable
> and hardcoded to 2*num_online_cpus. So any update of vm_commited_as by
> more than 256 pages will cause overflow in my test scenario which has
> 128 logical cpus.
> In the patch, I have set an enlargement multiplication factor for
> vm_commited_as's batch limit. I limit the sum of all local counters up
> to 5% of the total pages before overflowing into the global counter.
> This will avoid the frequent contention of the spin_lock in
> vm_commited_as. Some additional work will need to be done to make
> setting of this multiplication factor cpu hotplug aware. Advise on
> better approaches are welcomed.
> ...
> Signed-off-by: Tim Chen <>
> diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h
> index 46f6ba5..5a892d8 100644
> --- a/include/linux/percpu_counter.h
> +++ b/include/linux/percpu_counter.h
> @@ -21,6 +21,7 @@ struct percpu_counter {
> struct list_head list; /* All percpu_counters are on a list */
> #endif
> + u32 multibatch;
> s32 __percpu *counters;
> };

I dunno. Wouldn't it be better to put a `batch' field into
percpu_counter and then make the global percpu_counter_batch just go

That would require modifying each counter's `batch' at cpuhotplug time,
while somehow retaining the counter's user's intent. So perhaps the
counter would need two fields - original_batch and operating_batch or

 \ /
  Last update: 2011-01-28 00:39    [W:0.045 / U:0.204 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site