lkml.org 
[lkml]   [2014]   [Oct]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: RCU fanout leaf balancing
On 25/10/2014 21:48, Paul E. McKenney wrote:
> On Sat, Oct 25, 2014 at 02:59:24PM +0800, Daniel J Blueman wrote:
>> Hi Paul,
>>
>> Finding earlier reference to increasing RCU fanout leaf for the
>> purpose of "decrease[ing] cache-miss overhead for large systems",
>> would your suggestion be to increase the value to the next hierarchy
>> core-count above 16?
>>
>> If we have say 32 interconnected 48-core servers; 3 sockets of
>> dual-node 8-core Opteron 6300s, so 1536 cores in all. Latency across
>> the coherent interconnect is O(100x) higher than the internal
>> Hypertransport interconnect, so if we set RCU_FANOUT_LEAF to 48 to
>> keep leaf-checking local to one Hypertransport fabric, what wisdom
>> would one use for RCU_FANOUT? 4x leaf?
>>
>> Or, would it be more cache-friendly to set RCU_FANOUT_LEAF to 8 and
>> RCU_FANOUT to 48?
>
> The easiest approach would be to use the default of 16. Assuming
> consecutive CPU numbering within each 48-core server, this would mean that
> you would have three rcu_node structures per 48-core server. The next
> level up would of course span servers, but that level is accessed much
> less frequently than is the root level, so this should still work.
>
> If you also have hyperthreading, so that there are 96 hardware threads
> per server, and if you are using the same "interesting" numbering scheme
> that Intel uses, then this still works. You have three leaf rcu_node
> structure for the first set of hardware threads and another set of three
> for the second set of hardware threads.
>
> Or are you seeing some problem with the default? If so, please tell me
> what that problem is.
>
> You can of course increase RCU_FANOUT to 24 or 48 (this latter assuming
> a 64-bit kernel), at least if you are using a recent kernel. However,
> the penalty for too large a value for RCU_FANOUT is lock contention at
> scheduling-clock-interrupt time. So if you are setting RCU_FANOUT to 48,
> you probably also want to boot with skew_tick set.
>
> But the best approach is to try it. I bet that the default will work
> just fine for you. ;-)

Good info. I'll stick with the defaults of 16/64, will schedule some
tuning later, and let you know if I find anything significant.

Thanks Paul!

Daniel
--
Daniel J Blueman
Principal Software Engineer, Numascale


\
 
 \ /
  Last update: 2014-10-27 02:41    [W:0.140 / U:0.540 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site