lkml.org 
[lkml]   [2014]   [Jul]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
From
Subject[PATCH 05/10] smp: Fast path check on IPI list
Date
When we enqueue a remote irq work, we trigger the same IPI as those
raised by smp_call_function_*() family.

So when we receive such IPI, we check both irq_work and smp_call_function
queues. Thus if we trigger a remote irq work, we'll likely find the
smp_call_function queue empty unless we collide with concurrent enqueuers
but the probability is low.

Meanwhile, checking the smp_call_function queue can be costly because
we use llist_del_all() which relies on cmpxchg().

We can reduce this overhead by doing a fast path check with llist_empty().
Given the implicit IPI ordering:

Enqueuer Dequeuer
--------- --------
llist_add(csd, queue) get_IPI() {
send_IPI() if (llist_empty(queue)
...
When the IPI is sent, we are guaranteed that the IPI receiver will
see the new csd.

So lets do the fast path check to optimize non smp_call_function() related
jobs.

Cc: Ingo Molnar <mingo@kernel.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
kernel/smp.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index a1812d1..34378d4 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -184,11 +184,19 @@ static int generic_exec_single(int cpu, struct call_single_data *csd,
*/
void generic_smp_call_function_single_interrupt(void)
{
+ struct llist_head *head = &__get_cpu_var(call_single_queue);
struct llist_node *entry;
struct call_single_data *csd, *csd_next;
static bool warned;

- entry = llist_del_all(&__get_cpu_var(call_single_queue));
+ /*
+ * Fast check: in case of irq work remote queue, the IPI list
+ * is likely empty. We can spare the expensive llist_del_all().
+ */
+ if (llist_empty(head))
+ goto irq_work;
+
+ entry = llist_del_all(head);
entry = llist_reverse_order(entry);

/*
@@ -212,6 +220,7 @@ void generic_smp_call_function_single_interrupt(void)
csd_unlock(csd);
}

+irq_work:
/*
* Handle irq works queued remotely by irq_work_queue_on().
* Smp functions above are typically synchronous so they
--
1.8.3.1


\
 
 \ /
  Last update: 2014-07-19 10:21    [W:0.176 / U:0.388 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site