lkml.org 
[lkml]   [2010]   [Nov]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path
    On 10/27/10 12:44, Seiji Aguchi wrote:
    > Hi,
    >
    >> What actual problem are we solving here? Why is the current code
    >> inadequate? It would help to demonstrate some use-case and to explain
    >> how the situation improved with this patch.
    >
    > [Purpose]
    > My purpose is developing highly reliable logging facility for
    enterprise use.
    >
    > I'm planning to add the following triggers of kmsg_dumper().
    > - reboot/poweroff/halt/emergency_restart (this patch)
    > - Machine check
    >
    > I'm also planning to add an feature outputting kernel messages to
    NVRAM,
    > because NVRAM is equipped with enterprise servers.
    > We can realize highly reliable logging facility by outputting
    kernel messages to NVRAM.
    > (NVRAM is commonly used on Mainframe and Commercial Unix as well.)
    >
    > [Use case of reboot/poweroff/halt/emergency_restart]
    >
    > My company has often experienced the followings in our support service.
    > - Customer's system suddenly reboots.
    > - Customers ask us to investigate the reason of the reboot.
    >
    > We recognize the fact itself because boot messages remain in
    /var/log/messages.
    > However, we can't investigate the reason why the system rebooted,
    > because the last messages don't remain.
    > And off course we can't explain the reason.
    >
    >
    > We can solve above problem with this patch as follows.
    > Case1: reboot with command
    > - We can see "Restarting system with command:" or ""Restarting
    system.".
    >
    > Case2: halt with command
    > - We can see "System halted.".
    >
    > Case3: poweroff with command
    > - We can see " Power down.".
    >
    > Case4: emergency_restart with sysrq.
    > - We can see "Sysrq:" outputted in __handle_sysrq().
    >
    > Case5: emergency_restart with softdog.
    > - We can see "Initiating system reboot" in watchdog_fire().
    >
    > So, we can distinguish the reason of reboot, poweroff, halt and
    emergency_restart.
    >
    > If customer executed reboot command, you may think the customer
    should know the fact.
    > However, they often claim they don't execute the command when they
    rebooted system by mistake.
    >
    > No evidential message remain on current Linux kernel, so we can't
    show the proof to the customer.
    > This patch improves this situation.
    >
    > Seiji

    We carry patches in our kernels that do very similar things. The reason
    is essentially the same as what you have cited. On our platforms we have
    two different ways of storing events to an event log. One communicates
    with the BIOS itself; the other writes bit flags to a known area of
    non-volatile storage. That way when the machine comes back up we have a
    clear eventlog (with times) as to what happened when. Piecing these
    events together has proven to be invaluable for finding issues.

    For both of the drivers that log these events they use a shared
    interface that collect various events in the kernel and present them
    through a single notifier chain for the drivers' consumption.

    The things we currently track and log are the following:
    - clean reboot/shutdown
    - panic
    - oops
    - die
    - NMI watchdog

    An example eventlog produced by our systems looks like the following
    (63-67 are the boot numbers of the system in question):

    2010-10-14 10:26:06 | System Reset | 63
    2010-10-14 10:26:19 | System boot | 63
    2010-10-14 11:36:43 | Kernel Shutdown | 63 | Unknown Shutdown Reason
    2010-10-14 11:36:43 | System Reset | 64
    2010-10-14 11:36:56 | System boot | 64
    2010-10-18 14:51:54 | Kernel Shutdown | 64 | Clean
    2010-10-18 14:52:38 | System Reset | 65
    2010-10-18 14:52:51 | System boot | 65
    2010-10-26 02:44:48 | Kernel Shutdown | 65 | Oops
    2010-10-26 02:44:48 | Kernel Shutdown | 65 | Die
    2010-10-26 02:44:49 | Kernel Shutdown | 65 | Panic
    2010-10-26 02:45:43 | System Reset | 66
    2010-10-26 02:45:56 | System boot | 66
    2010-10-26 02:49:22 | Kernel Shutdown | 66 | Clean
    2010-10-26 02:50:05 | System Reset | 67
    2010-10-26 02:50:18 | System boot | 67
    2010-10-26 11:39:20 | Kernel Shutdown | 67 | Clean

    Hope that helps others know that we think such a mechansim is vital. I
    can post the patches for the common infrastructure if people are interested.

    -Aaron


    \
     
     \ /
      Last update: 2010-11-03 22:55    [W:0.031 / U:32.804 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site