Messages in this thread | | | Date | Fri, 18 Sep 2015 15:14:11 -0700 | From | Andrew Morton <> | Subject | Re: [PATCH 0/4] printk: Softlockup avoidance |
| |
On Wed, 19 Aug 2015 17:38:27 +0200 Jan Kara <jack@suse.com> wrote:
> From: Jan Kara <jack@suse.cz> > > Hello, > > since lately there were several attempts at dealing with softlockups due > to heavy printk traffic [1] [2] and I've been also privately pinged by > couple of people about the state of the patch set, I've decided to respin > the patch set. > > To remind the original problem: > > Currently, console_unlock() prints messages from kernel printk buffer to > console while the buffer is non-empty. When serial console is attached, > printing is slow and thus other CPUs in the system have plenty of time > to append new messages to the buffer while one CPU is printing. Thus the > CPU can spend unbounded amount of time doing printing in console_unlock(). > This is especially serious when printk() gets called under some critical > spinlock or with interrupts disabled. > > In practice users have observed a CPU can spend tens of seconds printing > in console_unlock() (usually during boot when hundreds of SCSI devices > are discovered) resulting in RCU stalls (CPU doing printing doesn't > reach quiescent state for a long time), softlockup reports (IPIs for the > printing CPU don't get served and thus other CPUs are spinning waiting > for the printing CPU to process IPIs), and eventually a machine death > (as messages from stalls and lockups append to printk buffer faster than > we are able to print). So these machines are unable to boot with serial > console attached. Also during artificial stress testing SATA disk > disappears from the system because its interrupts aren't served for too > long. > > This series addresses the problem in the following way: If CPU has printed > more that printk_offload (defaults to 1000) characters, it wakes up one > of dedicated printk kthreads (we don't use workqueue because that has > deadlock potential if printk was called from workqueue code). Once we find > out kthread is spinning on a lock, we stop printing, drop console_sem, and > let kthread continue printing. Since there are two printing kthreads, they > will pass printing between them and thus no CPU gets hogged by printing.
I still hate your patchset ;)
But nothing better suggests itself. I have a few review comments - please let's work through that stuff, get a fresh version out and we'll see how it goes.
Is this patchset being used in the field? Perhaps in the suse kernel? If so, a mention of that in the changelog would help things along.
| |