lkml.org 
[lkml]   [2014]   [Mar]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 5/6] printk: Hand over printing to console if printing too long
    On Thu 13-03-14 16:06:25, Andrew Morton wrote:
    > On Thu, 13 Mar 2014 16:58:37 +0100 Jan Kara <jack@suse.cz> wrote:
    >
    > > Currently, console_unlock() prints messages from kernel printk buffer to
    > > console while the buffer is non-empty. When serial console is attached,
    > > printing is slow and thus other CPUs in the system have plenty of time
    > > to append new messages to the buffer while one CPU is printing. Thus the
    > > CPU can spend unbounded amount of time doing printing in console_unlock().
    > > This is especially serious problem if the printk() calling
    > > console_unlock() was called with interrupts disabled.
    > >
    > > In practice users have observed a CPU can spend tens of seconds printing
    > > in console_unlock() (usually during boot when hundreds of SCSI devices
    > > are discovered) resulting in RCU stalls (CPU doing printing doesn't
    > > reach quiescent state for a long time), softlockup reports (IPIs for the
    > > printing CPU don't get served and thus other CPUs are spinning waiting
    > > for the printing CPU to process IPIs), and eventually a machine death
    > > (as messages from stalls and lockups append to printk buffer faster than
    > > we are able to print). So these machines are unable to boot with serial
    > > console attached. Also during artificial stress testing SATA disk
    > > disappears from the system because its interrupts aren't served for too
    > > long.
    > >
    > > This patch implements a mechanism where after printing specified number
    > > of characters (tunable as a kernel parameter printk.offload_chars), CPU
    > > doing printing asks for help by setting PRINTK_HANDOVER_B bit in
    > > printk_handover_state variable and wakes up one of dedicated kthreads.
    > > As soon as the printing CPU notices kthread got scheduled and is
    > > spinning on console_sem, it drops console_sem and exits
    > > console_unlock(). kthread then takes over printing instead. This way no
    > > CPU should spend printing too long even if there is heavy printk
    > > traffic.
    >
    > Sigh ;)
    >
    > Creating yet more kernel threads to solve this problem is a bit sad.
    > Is there no way in which we can borrow one of the existing threads?
    I realized I didn't react to the above. I've checked which kernel threads
    are running on my system and I can see:
    rcu related threads, ksoftirqd, various fs threads, kauditd - nothing
    really seems as a good fit. Then there are various workqueue related
    threads - these aren't really suitable either. Finally I can see migration
    threads (from kernel/stop_machine.c) - I'm not sure I want to mess with
    these - especially since they can run any function somebody passes to them
    and when that function calls printk() things might get interesting.

    So I would go for starting the threads only if printk.offload_chars ever
    gets set to value > 0.

    Honza
    --
    Jan Kara <jack@suse.cz>
    SUSE Labs, CR


    \
     
     \ /
      Last update: 2014-03-14 15:01    [W:2.215 / U:0.280 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site