Messages in this thread | | | Date | Thu, 2 Apr 2015 17:19:51 +0200 | From | Frederic Weisbecker <> | Subject | Re: [PATCH v2] watchdog: nohz: don't run watchdog on nohz_full cores |
| |
On Mon, Mar 30, 2015 at 04:02:06PM -0400, Don Zickus wrote: > On Mon, Mar 30, 2015 at 03:32:55PM -0400, Chris Metcalf wrote: > > On 03/30/2015 03:12 PM, Don Zickus wrote: > > >On Mon, Mar 30, 2015 at 02:51:05PM -0400, cmetcalf@ezchip.com wrote: > > >>From: Chris Metcalf <cmetcalf@ezchip.com> > > >> > > >>Running watchdog can be a helpful debugging feature on regular > > >>cores, but it's incompatible with nohz_full, since it forces > > >>regular scheduling events. Accordingly, just exit out immediately > > >>from any nohz_full core. > > >> > > >>An alternate approach would be to add a flags field or function to > > >>smp_hotplug_thread to control on which cores the percpu threads > > >>are created, but it wasn't clear that much mechanism was useful. > > >Hi Chris, > > > > > >It seems like the correct solution would be to hook into the idle_loop > > >somehow. If the cpu is idle, then it seems unlikely that a lockup could > > >occur. > > > > With nohz_full, though, the cpu might be running userspace code > > with the intention of keeping kernel ticks disabled. Even returning > > to kernel mode to try to figure out if we "should" be running the > > watchdog on a given core will induce exactly the kind of interrupts > > that nohz_full is designed to prevent. > > > > My assumption is generally that nohz_full cores don't spend a lot of > > time in the kernel anyway, as they are optimized for user space. > > > > I guess you could imagine doing something per-cpu on the nohz_full > > cores where we effectively call watchdog_disable() whenever a > > nohz_full core enters userspace, and watchdog_enable() whenever it > > enters the kernel. We could add some per-cpu state in the watchdog > > code to track whether that core was currently enabled or disabled > > to avoid double-enabling or double-disabling. I would think > > context_tracking_user_exit()/_enter() would be the place to do this. > > > > This feels like a lot of overhead, potentially. Thoughts? > > A few months ago I might have thought that a reasonable approach. But > recently we have added code to make the watchdog an all or nothing approach > across the system. This might make it difficult to do what you are > suggesting. > > I do not know enough about the nohz code to know what the right approach is > here. Perhaps Federic can enlighten me?
Well, cancelling/rearming a timer on every userspace round trip sounds way too much overhead to me :-)
But Ingo's suggestion to disable it properly (only on nohz full core) looks good. And we should be able to re-enable it everywhere with "sysctl -w kernel.watchdog=1" and you need to warn about this on boot.
| |