Messages in this thread | | | Date | Fri, 29 Jul 2011 16:53:21 +1000 | From | Greg Banks <> | Subject | Re: Fw: [PATCH] sunrpc: use better NUMA affinities |
| |
On 29/07/11 16:30, Eric Dumazet wrote: > Le vendredi 29 juillet 2011 à 16:05 +1000, Greg Banks a écrit : >> On 29/07/11 15:32, NeilBrown wrote: >> >> I seem to remember coming to the conclusion that Jeff eventually >> addressed this problem...am I misremembering or did something regress? >> > Currently, all nfsd kthreads use memory for their kernel stack and > various initial data from a _single_ node, even if you use > sunrpc.pool_mode=pernode (or percpu)
That's just plain broken and I'm very pleased to see you fix it.
I was just surprised that it was still broken and wondering how that happened. Looking at ToT I see that because I dropped the ball in 2008, Jeff's patches didn't address the problem. In ToT svc_pool_map_set_cpumask() is called *after* kthread_create() and applies to the child thread, *after* it's stack has been allocated on the wrong node. In the working SGI code, svc_pool_map_set_cpumask() is called by the parent node on itself *before* calling kernel_thread() or doing any of the data structure allocations, thus ensuring that everything gets allocated using the default memory allocation policy, which on SGI NFS servers was globally tuned to be "node-local".
> With my patch, we make sure each thread gets its stack from its local > node. > > Check commit 94dcf29a11b3d20a (kthread: use kthread_create_on_node()) to > see how this strategy already was adopted for ksoftirqd, kworker, > migration, and pktgend kthreads.
Ah, I see. It's unfortunate that the kthread_create() API ends up being passed a CPU number but that's only used to format the name and not for sensible things :(
-- Greg.
-- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |