[lkml]   [2010]   [Feb]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectNO_HZ migration of TCP ack timers


    We have a networking workload on a large ppc64 box that is spending a lot
    of its time in mod_timer(). One backtrace looks like:

    83.25% [k] ._spin_lock_irqsave
    |--99.62%-- .lock_timer_base
    | .mod_timer
    | .sk_reset_timer
    | |
    | |--84.77%-- .tcp_send_delayed_ack
    | | .__tcp_ack_snd_check
    | | .tcp_rcv_established
    | | .tcp_v4_do_rcv

    | |--12.72%-- .tcp_ack
    | | .tcp_rcv_established
    | | .tcp_v4_do_rcv

    So it's mod_timer being called from the TCP ack timer code. It looks like
    commit eea08f32adb3f97553d49a4f79a119833036000a (timers: Logic to move non
    pinned timers) is causing it, in particular:

    #if defined(CONFIG_NO_HZ) && defined(CONFIG_SMP)
    if (!pinned && get_sysctl_timer_migration() && idle_cpu(cpu)) {
    int preferred_cpu = get_nohz_load_balancer();

    if (preferred_cpu >= 0)
    cpu = preferred_cpu;


    echo 0 > /proc/sys/kernel/timer_migration

    makes the problem go away.

    I think the problem is the CPU is most likely to be idle when an rx networking
    interrupt comes in. It seems the wrong thing to do to migrate any ack timers
    off the current cpu taking the interrupt, and with enough networks we train
    wreck transferring everyones ack timers to the nohz load balancer cpu.

    What should we do? Should we use mod_timer_pinned here? Or is this an issue
    other areas might see (eg the block layer) and we should instead avoid
    migrating timers created out of interrupts.


     \ /
      Last update: 2010-02-18 06:31    [W:0.019 / U:1.092 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site