[lkml]   [2008]   [Oct]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: NMI watchdog setup_lapic_nmi_watchdog() problem

    On Tue, Oct 7, 2008 at 10:21 PM, Andrew Morton
    <> wrote:
    > On Tue, 7 Oct 2008 17:45:32 +0200
    > "stephane eranian" <> wrote:
    >> Hello,
    >> I was doing some more testing with perfmon when I ran into
    >> a problem with the NMI watchdog code in 2.6.27-rc8.
    >> Since 2.6.20, it is possible to enable/disable the NMI watchdog
    >> on-the-fly via /proc/sys/kernel/nmi_watchdog. This is a nice option
    >> which avoids having to reboot the kernel.
    >> Enabling/disabling the NMI watchdog uses two internal functions
    >> enable_lapic_nmi_watchdog() and disable_lapic_nmi_watchdog().
    >> Enable_lapic_nmi_watchdog() uses a IPI handler to setup the
    >> APIC on each CPU. However, it turns out that this handler, namely,
    >> setup_apic_nmi_watchdog() relies on some explicit ordering constraint
    >> due to suspend/resume constraints as explained in the comment
    >> below:
    >> void setup_apic_nmi_watchdog(void *unused)
    >> {
    >> if (__get_cpu_var(wd_enabled))
    >> return;
    >> /* cheap hack to support suspend/resume */
    >> /* if cpu0 is not active neither should the other cpus */
    >> if (smp_processor_id() != 0 && atomic_read(&nmi_active) <= 0)
    >> return;
    >> switch (nmi_watchdog) {
    >> [snip]
    >> }
    >> Supposing watchdog was disabled via /proc, nmi_active = 0. Then if you
    >> re-enable, and if CPU0 is not the first to execute the IPI handler, then none
    >> of the other CPUS will re-enable their NMI watchdog timer. On a quad-core
    >> system, I have seen, for instance, 2 out of 4 with NMI watchdogs re-enabled.
    >> I am not an expert at suspend/resume. I am assuming there was a race condition
    >> there and that's why this code was added early on. The problem is that it now
    >> conflicts with the /proc option.
    >> It is not clear to me how this works during boot. Obviously the order
    >> is respected
    >> and all CPUs have their NMI watchdog enabled.
    >> Until I understand the suspend/resume issue, it is hard to provide a
    >> fix for this.
    >> Any comments?
    > The "cheap hack" was added in September 2006.
    > 2.6.20 was released in Feb 2007.
    > So presumably this problem has always been there, since
    > /proc/sys/kernel/nmi_watchdog was first added, only nobody has hit it
    > before.
    Yes, I believe that is true. The bug was there as soon as the /proc interface
    was introduced.

    The bug is not visible unless you instrument that setup_lapic_nmi_watchdog()
    routine. I did that because I was tracking the value of nmi_active
    within perfmon.
    On a quad-core, nmi_active is equal to 4, so when I saw 2, I started

    > Have you only recently started to use /proc/sys/kernel/nmi_watchdog, or
    > did it work OK on any earlier kernel?
    I started playing with that because wanted to see whether NMI was releasing
    the PMU MSR back to the free pool which it did. However, I had not
    paid attention
    to whether or not NMI was re-activated on all CPUs.

    If you remove the 'cheap hack', disabling/enabling works. That's why
    I'd like to better
    understand was is going on with suspend/resume.

     \ /
      Last update: 2008-10-07 23:13    [W:0.030 / U:7.528 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site