Messages in this thread Patch in this message | | | Subject | Re: [patch 7/7] sched: only use TTWU_QUEUE when waker/wakee CPUs do not share top level cache | From | Peter Zijlstra <> | Date | Wed, 23 Nov 2011 13:08:16 +0100 |
| |
I changed this a little, no point in iterating the domains when SD_SHARE_PKG_RESOURCES fails.
--- Subject: sched: Only use TTWU_QUEUE when waker/wakee CPUs do not share top level cache From: Mike Galbraith <mgalbraith@suse.de> Date: Tue, 22 Nov 2011 15:26:25 +0100
TTWU_QUEUE IPI overhead was measured to be as much as 13% of netperf TCP_RR overhead when waking to a shared cache. Don't IPI unless we're waking cross cache, where it can be a winner.
Cc: Chris Mason <chris.mason@oracle.com> Cc: Dave Kleikamp <dave.kleikamp@oracle.com> Suggested-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Signed-off-by: Mike Galbraith <efault@gmx.de> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/1321971985.6855.26.camel@marge.simson.net --- kernel/sched/core.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)
Index: linux-2.6/kernel/sched/core.c =================================================================== --- linux-2.6.orig/kernel/sched/core.c +++ linux-2.6/kernel/sched/core.c @@ -1480,12 +1480,36 @@ static int ttwu_activate_remote(struct t #endif /* __ARCH_WANT_INTERRUPTS_ON_CTXSW */ #endif /* CONFIG_SMP */ +static int ttwu_share_cache(int this_cpu, int cpu) +{ +#ifndef CONFIG_X86 + struct sched_domain *sd; + int ret = 0; + + rcu_read_lock(); + for_each_domain(this_cpu, sd) { + if (!(sd->flags & SD_SHARE_PKG_RESOURCES)) + break; + + if (cpumask_test_cpu(cpu, sched_domain_span(sd))) { + ret = 1; + break; + } + } + rcu_read_unlock(); + + return ret; +#else + return per_cpu(cpu_llc_id, this_cpu) == per_cpu(cpu_llc_id, cpu); +#endif +} + static void ttwu_queue(struct task_struct *p, int cpu) { struct rq *rq = cpu_rq(cpu); #if defined(CONFIG_SMP) - if (sched_feat(TTWU_QUEUE) && cpu != smp_processor_id()) { + if (sched_feat(TTWU_QUEUE) && !ttwu_share_cache(smp_processor_id(), cpu)) { sched_clock_cpu(cpu); /* sync clocks x-cpu */ ttwu_queue_remote(p, cpu); return;
| |