[lkml]   [2010]   [Feb]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectNO_HZ migration of TCP ack timers


We have a networking workload on a large ppc64 box that is spending a lot
of its time in mod_timer(). One backtrace looks like:

83.25% [k] ._spin_lock_irqsave
|--99.62%-- .lock_timer_base
| .mod_timer
| .sk_reset_timer
| |
| |--84.77%-- .tcp_send_delayed_ack
| | .__tcp_ack_snd_check
| | .tcp_rcv_established
| | .tcp_v4_do_rcv

| |--12.72%-- .tcp_ack
| | .tcp_rcv_established
| | .tcp_v4_do_rcv

So it's mod_timer being called from the TCP ack timer code. It looks like
commit eea08f32adb3f97553d49a4f79a119833036000a (timers: Logic to move non
pinned timers) is causing it, in particular:

#if defined(CONFIG_NO_HZ) && defined(CONFIG_SMP)
if (!pinned && get_sysctl_timer_migration() && idle_cpu(cpu)) {
int preferred_cpu = get_nohz_load_balancer();

if (preferred_cpu >= 0)
cpu = preferred_cpu;


echo 0 > /proc/sys/kernel/timer_migration

makes the problem go away.

I think the problem is the CPU is most likely to be idle when an rx networking
interrupt comes in. It seems the wrong thing to do to migrate any ack timers
off the current cpu taking the interrupt, and with enough networks we train
wreck transferring everyones ack timers to the nohz load balancer cpu.

What should we do? Should we use mod_timer_pinned here? Or is this an issue
other areas might see (eg the block layer) and we should instead avoid
migrating timers created out of interrupts.


 \ /
  Last update: 2010-02-18 06:31    [W:0.056 / U:5.332 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site