lkml.org 
[lkml]   [2020]   [Jan]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC PATCH] PCI, kdump: Clear bus master bit upon shutdown in kdump kernel
    On 01/11/20 at 08:45am, Baoquan He wrote:
    > On 01/10/20 at 04:00pm, Jerry Hoemann wrote:
    > > > I am not understanding this failure mode either. That code in
    > > > pci_device_shutdown() was added originally to address this very issue.
    > > > The patch 4fc9bbf98fd6 ("PCI: Disable Bus Master only on kexec reboot")
    > > > shut down any errant DMAs from PCI devices as we kexec a new kernel. In
    > > > this new patch, this is the same code path that will be taken again when
    > > > kdump kernel is shutting down. If the errant DMA problem was not fixed
    > > > by clearing Bus Master bit in this path when kdump kernel was being
    > > > kexec'd, why does the same code path work the second time around when
    > > > kdump kernel is shutting down? Is there more going on that we don't
    > > > understand?
    > > >
    > >
    > > Khalid,
    > >
    > > I don't believe we execute that code path in the crash case.
    > >
    > > The variable kexec_in_progress is set true in kernel_kexec() before calling
    > > machine_kexec(). This is the fast reboot case.
    > >
    > > I don't see kexec_in_progress set true elsewhere.
    > >
    > >
    > > The code path for crash is different.
    > >
    > > For instance, panic() will call
    > > -> __crash_kexec() which calls
    > > -> machine_kexec().
    > >
    > > So the setting of kexec_in_progress is bypassed.
    >
    > Yeah, it's a differet behaviour than kexec case. I talked to Kairui, the
    > patch log may be not very clear. Below is summary I got from my
    > understanding about this issue:
    >
    > ~~~~~~~~~~~~~~~~~~~~~~~
    > Problem:
    >
    > When crash is triggered, system jumps into kdump kernel to collect
    > vmcore and dump out. After dumping is finished, kdump kernel will try
    > ty reboot to normal kernel. This hang happened during kdump kernel
    > rebooting, when dumping is network dumping, e.g ssh/nfs, local storage
    > is HPSA.
    >
    > Root cause:
    >
    > When configuring network dumping, only network driver modules are added
    > into kdump initramfs. However, the storage HPSA pcie device is enabled
    > in 1st kernel, its status is PCI_D3hot. When crashed system jumps to kdump
    > kernel, we didn't shutdown any device for safety and efficiency. Then
    > during kdump kernel boot up, the pci scan will get hpsa device and only
    > initialize its status as pci_dev->current_state = PCI_UNKNOWN. This
    > pci_dev->current_state will be manipulated by the relevant device
    > driver. So HPSA device will never have chance to calibrate its status,
    > and can't be shut down by pci_device_shutdown() called by reboot
    > service. It's still PCI_D3hot, then crash happened when system try to
    ~~~~~ s/crash/hang/, sorry, typo again
    > shutdown its upper bridge.
    >
    > Fix:
    >
    > Here, Kairui uses a quirk to get PM state and mask off value bigger than
    > PCI_D3cold. Means, all devices will get PM state
    > pci_dev->current_state = PCI_D0 or PCI_D3hot. Finally, during kdump
    > reboot stage, this device can be shut down successfully by clearing its
    > master bit.
    >
    > ~~~~~~~~~~~~~~~
    >
    > About this patch, I think the quirk getting active PM state for all devices
    > may be risky, it will impact normal kernel too which doesn't have this issue.
    >
    > Wondering if there's any other way to fix or work around it.
    >
    > Thanks
    > Baoquan

    \
     
     \ /
      Last update: 2020-01-11 02:47    [W:2.637 / U:0.212 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site