Messages in this thread |  | | Date | Wed, 22 Jul 2015 16:29:46 -0700 | | From | Jörn Engel <> | | Subject | Re: [PATCH] soft lockup: kill realtime threads before panic |
| |
On Wed, Jul 22, 2015 at 03:54:36PM -0700, Andrew Morton wrote: > On Tue, 21 Jul 2015 15:07:57 -0700 Spencer Baugh <sbaugh@catern.com> wrote: > > > From: Joern Engel <joern@logfs.org> > > > > We have observed cases where the soft lockup detector triggered, but no > > kernel bug existed. Instead we had a buggy realtime thread that > > monopolized a cpu. So let's kill the responsible party and not panic > > the entire system. > > > > ... > > > > --- a/kernel/watchdog.c > > +++ b/kernel/watchdog.c > > @@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) > > } > > > > add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); > > - if (softlockup_panic) > > + if (rt_prio(current->prio)) { > > + pr_emerg("killing realtime thread\n"); > > + send_sig(SIGILL, current, 0); > > Why choose SIGILL?
It is a random signal that happens to generate a stacktrace in userspace.
> > + } else if (softlockup_panic) > > panic("softlockup: hung tasks"); > > __this_cpu_write(soft_watchdog_warn, true); > > But what about a non-buggy realtime thread which happens to > occasionally spend 15 seconds doing stuff? > > Old behaviour: kernel blurts a softlockup message, everything keeps running. > > New behaviour: thread gets killed, plane crashes. > > > Possibly a better approach would be to only kill the thread if > softlockup_panic was set, because the system is going down anyway. > > Also, perhaps some users would prefer that the kernel simply suppress > the softlockup warning in this situation, rather than killing stuff! > > > > > Really, what you're trying to implement here is a watchdog for runaway > realtime threads. And that sounds a worthy project but it's a rather > separate thing from the softlockup detector. A realtime thread > watchdog feature might have things as > > - timeout duration separately configurable from softlockup > > - enabled independently from sotflockup: people might want one and > not the other. > > - configurable signal, perhaps? > > Now, the *implementation* of the realtime thread watchdog may well > share code with the softlockup detector. But from a > conceptual/configuration/documentation point of view, it's a separate > thing, no?
Agreed. We needed this patch exactly once and it is a rather quick hack that yielded the necessary results. Realtime threads were well-behaved since and the patch has seen zero polish as a result.
I think it is better to drop the patch for now. If someone else keeps running into the same issue, it might be a starting point for a better implementation. They will find it in list archives.
Jörn
-- I can say that I spend most of my time fixing bugs even if I have lots of new features to implement in mind, but I give bugs more priority. -- Andrea Arcangeli, 2000
|  |