lkml.org 
[lkml]   [2012]   [Jan]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    SubjectRe: [PATCH] x86, kdump, ioapic: Fix kdump race with migrating irq
    Date
    Don Zickus <dzickus@redhat.com> writes:

    > A customer of ours noticed when their machine crashed, kdump did not
    > work but hung instead. Using their firmware dumping solution they
    > grabbed a vmcore and decoded the stacks on the cpus. What they
    > noticed seemed to be a rare deadlock with the ioapic_lock.
    >
    > CPU4:
    > machine_crash_shutdown
    > -> machine_ops.crash_shutdown
    > -> native_machine_crash_shutdown
    > -> kdump_nmi_shootdown_cpus ------> Send NMI to other CPUs
    > -> disable_IO_APIC
    > -> clear_IO_APIC
    > -> clear_IO_APIC_pin
    > -> ioapic_read_entry
    > -> spin_lock_irqsave(&ioapic_lock, flags)
    > ---Infinite loop here---
    >
    > CPU0:
    > do_IRQ
    > -> handle_irq
    > -> handle_edge_irq
    > -> ack_apic_edge
    > -> move_native_irq
    > -> mask_IO_APIC_irq
    > -> mask_IO_APIC_irq_desc
    > -> spin_lock_irqsave(&ioapic_lock, flags)
    > ---Receive NMI here after getting spinlock---
    > -> nmi
    > -> do_nmi
    > -> crash_nmi_callback
    > ---Infinite loop here---
    >
    > The problem is that although kdump tries to shutdown minimal hardware,
    > it still needs to disable the IO APIC. This requires spinlocks which
    > may be held by another cpu. This other cpu is being held infinitely in
    > an NMI context by kdump in order to serialize the crashing path. Instant
    > deadlock.

    Can you test to see if kexec on panic still needs to disable the IO
    APIC. Last I looked we were close if not all of the way there to not
    needing to boot the kernel in pic mode?

    If we can skip the ioapic disable entirely we should be much more
    robust.

    Eric


    \
     
     \ /
      Last update: 2012-01-31 23:09    [W:0.025 / U:60.868 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site