lkml.org 
[lkml]   [2015]   [Nov]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [RFC] mempolicy: convert the shared_policy lock to a rwlock
On Thu, 12 Nov 2015, Nathan Zimmer wrote:

> When running the SPECint_rate gcc on some very large boxes it was noticed
> that the system was spending lots of time in mpol_shared_policy_lookup.
> The gamess benchmark can also show it and is what I mostly used to chase
> down the issue since the setup for that I found a easier.
>
> To be clear the binaries were on tmpfs because of disk I/O reqruirements.
> We then used text replication to avoid icache misses and having all the
> copies banging on the memory where the instruction code resides.
> This results in us hitting a bottle neck in mpol_shared_policy_lookup
> since lookup is serialised by the shared_policy lock.
>
> I have only reproduced this on very large (3k+ cores) boxes. The problem
> starts showing up at just a few hundred ranks getting worse until it
> threatens to livelock once it gets large enough.
> For example on the gamess benchmark at 128 ranks this area consumes only
> ~1% of time, at 512 ranks it consumes nearly 13%, and at 2k ranks it is
> over 90%.
>
> To alleviate the contention on this area I converted the spinslock to a
> rwlock. This allows the large number of lookups to happen simultaneously.
> The results were quite good reducing this to consumtion at max ranks to
> around 2%.
>

There're a couple of places in the sp_lookup() comment that would need to
be fixed to either correct that this is no longer a spinlock and that the
caller must hold the read lock. The comment for sp_insert() would have to
be fixed to specify the caller must hold the write lock. When that's
fixed, feel free to add

Acked-by: David Rientjes <rientjes@google.com>


\
 
 \ /
  Last update: 2015-11-12 22:41    [W:0.469 / U:0.052 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site