lkml.org 
[lkml]   [2015]   [Oct]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH 3/7] kernel: Avoid softlockups in stop_machine() during heavy printing
  Hmph, sorry for the x/7 numbering. Patch 7 was the debug patch which I
didn't send...

Honza

On Mon 26-10-15 05:52:46, Jan Kara wrote:
> From: Jan Kara <jack@suse.cz>
>
> When there are lots of messages accumulated in printk buffer, printing
> them (especially over serial console) can take a long time (tens of
> seconds). stop_machine() will effectively make all cpus spin in
> multi_cpu_stop() waiting for the CPU doing printing to print all the
> messages which triggers NMI softlockup watchdog and RCU stall detector
> which add even more to the messages to print. Since machine doesn't do
> anything (except serving interrupts) during this time, also network
> connections are dropped and other disturbances may happen.
>
> Paper over the problem by waiting for printk buffer to be empty before
> starting to stop CPUs. In theory a burst of new messages can be appended
> to the printk buffer before CPUs enter multi_cpu_stop() so this isn't a 100%
> solution but it works OK in practice and I'm not aware of a reasonably
> simple better solution.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> include/linux/console.h | 11 +++++++++++
> kernel/printk/printk.c | 25 +++++++++++++++++++++++++
> kernel/stop_machine.c | 9 +++++++++
> 3 files changed, 45 insertions(+)
>
> diff --git a/include/linux/console.h b/include/linux/console.h
> index bd194343c346..96da462cdfeb 100644
> --- a/include/linux/console.h
> +++ b/include/linux/console.h
> @@ -150,6 +150,17 @@ extern int console_trylock(void);
> extern void console_unlock(void);
> extern void console_conditional_schedule(void);
> extern void console_unblank(void);
> +#ifdef CONFIG_SMP
> +extern void printk_log_buf_drain(void);
> +#else
> +/*
> + * In non-SMP kernels there won't be much to drain so save some code for tiny
> + * kernels.
> + */
> +static inline void printk_log_buf_drain(void)
> +{
> +}
> +#endif
> extern struct tty_driver *console_device(int *);
> extern void console_stop(struct console *);
> extern void console_start(struct console *);
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index b9bb4a7a6dff..8dc6c146d022 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -2488,6 +2488,31 @@ struct tty_driver *console_device(int *index)
> return driver;
> }
>
> +/* For non-SMP kernels this function isn't used and would be pointless anyway */
> +#ifdef CONFIG_SMP
> +/*
> + * Wait until all messages accumulated in the printk buffer are printed to
> + * console. Note that as soon as this function returns, new messages may be
> + * added to the printk buffer by other CPUs.
> + */
> +void printk_log_buf_drain(void)
> +{
> + bool retry;
> + unsigned long flags;
> +
> + while (1) {
> + raw_spin_lock_irqsave(&logbuf_lock, flags);
> + retry = console_seq != log_next_seq;
> + raw_spin_unlock_irqrestore(&logbuf_lock, flags);
> + if (!retry || console_suspended)
> + break;
> + /* Cycle console_sem to wait for outstanding printing */
> + console_lock();
> + console_unlock();
> + }
> +}
> +#endif
> +
> /*
> * Prevent further output on the passed console device so that (for example)
> * serial drivers can disable console output before suspending a port, and can
> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index 12484e5d5c88..e9496b4a3825 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -21,6 +21,7 @@
> #include <linux/smpboot.h>
> #include <linux/atomic.h>
> #include <linux/lglock.h>
> +#include <linux/console.h>
>
> /*
> * Structure to determine completion condition and record errors. May
> @@ -543,6 +544,14 @@ static int __stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cp
> return ret;
> }
>
> + /*
> + * If there are lots of outstanding messages, printing them can take a
> + * long time and all cpus would be spinning waiting for the printing to
> + * finish thus triggering NMI watchdog, RCU lockups etc. Wait for the
> + * printing here to avoid these.
> + */
> + printk_log_buf_drain();
> +
> /* Set the initial state and stop all online cpus. */
> set_state(&msdata, MULTI_STOP_PREPARE);
> return stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
> --
> 2.1.4
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR


\
 
 \ /
  Last update: 2015-10-26 06:21    [W:0.071 / U:0.500 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site