Messages in this thread | | | Date | Wed, 14 Jul 2010 19:01:20 -0700 | From | Yinghai Lu <> | Subject | Re: tip/master broken with x2apic and kexec |
| |
On 07/14/2010 05:03 PM, Suresh Siddha wrote: > On Wed, 2010-07-14 at 15:57 -0700, Yinghai Lu wrote: >> On 07/14/2010 02:23 PM, Yinghai Lu wrote: >>> On 07/14/2010 01:35 PM, Yinghai Lu wrote: >>>> On 07/13/2010 04:27 PM, Yinghai Lu wrote: >>>>> On 07/13/2010 03:00 PM, H. Peter Anvin wrote: >>>>>> On 07/12/2010 07:59 PM, Yinghai Lu wrote: >>>>>>> tip/master: >>>>>>> system1: BIOS enabled x2apic, first kernel boot well, and when kexec second kernel will cause system instant reboot. >>>>>>> >>>>>>> system2: BIOS not enable x2apic, first kernel boot well and enable x2apic, and kexec second kernel well. but when kexec third kernel will case system instant reboot. >>>>>>> >>>>>>> linus' tree is ok. >>>>>>> >>>>>>> but for system2 if boot with nox2apic ,intr-remaping off, iommu off, the kexec loop test will pass. >>>>>>> >>>>>>> the problem looks start in recent two or three weeks. >>>>>>> >>>>>>> Any idea? >>>>>>> >>>>>>> bisecting will take a while, because the system post take a while everytime. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Yinghai Lu >>>>>> >>>>>> OK, I found the bug... if you could test out the patch which will be >>>>>> sent out shortly I would very much appreciate it. >>>>> >>>>> not sure if your patch is the offending one now. >>>>> >>>>> kL: kernel from linus tree >>>>> kT1: kernel from tip >>>>> kT2: kernel from tip with reverting your patch >>>>> >>>>> BIOS-->kL ---> kL ---> kL....always working >>>>> BIOS-->kT1 ---> kT1 ---> kT1 : between second one and third one system reset instant... >>>>> BIOS-->kT2 ---> kT2 ---> kT2 : between second one and third one system reset instant... >>>>> >>>>> BIOS-->kL ---> kL ---> kL ---> then kT1 ---> kT1 .... always working >>>>> BIOS-->kL ---> kL ---> kL ---> then kT2 ---> kT2 .... always working >>>>> >>>> >>>> bisecting said: >>>> >>>>> git bisect good >>>> 58687acba59266735adb8ccd9b5b9aa2c7cd205b is the first bad commit >>>> commit 58687acba59266735adb8ccd9b5b9aa2c7cd205b >>>> Author: Don Zickus <dzickus@redhat.com> >>>> Date: Fri May 7 17:11:44 2010 -0400 >>>> >>>> lockup_detector: Combine nmi_watchdog and softlockup detector >>>> >>>> The new nmi_watchdog (which uses the perf event subsystem) is very >>>> similar in structure to the softlockup detector. Using Ingo's >>>> suggestion, I combined the two functionalities into one file: >>>> kernel/watchdog.c. >>>> >>>> Now both the nmi_watchdog (or hardlockup detector) and softlockup >>>> detector sit on top of the perf event subsystem, which is run every >>>> 60 seconds or so to see if there are any lockups. >>>> >>>> To detect hardlockups, cpus not responding to interrupts, I >>>> implemented an hrtimer that runs 5 times for every perf event >>>> overflow event. If that stops counting on a cpu, then the cpu is >>>> most likely in trouble. >>>> >>>> To detect softlockups, tasks not yielding to the scheduler, I used the >>>> previous kthread idea that now gets kicked every time the hrtimer fires. >>>> If the kthread isn't being scheduled neither is anyone else and the >>>> warning is printed to the console. >>>> >>>> I tested this on x86_64 and both the softlockup and hardlockup paths >>>> work. >>>> >>> >>> with >>> # CONFIG_LOCKUP_DETECTOR is not set >>> # CONFIG_HARDLOCKUP_DETECTOR is not set >>> >>> kexec loop test could passed. >>> >>> also that patch will break x2apic preenabled system 's kexec/kdump. >> >> before the combining patch >> >> CONFIG_DETECT_SOFTLOCKUP=y >> CONFIG_NMI_WATCHDOG=y >> >> will have the same problem. >> >> so the problem should come from NMI_WATCHDOG. > > Yinghai, It looks like some timing issue wrt nmi handling/kexec and > perhaps not directly related to x2apic? Perhaps we should try with > x2apic disabled but with intr-remapping enabled etc to see if it changes > anything.
only have "nox2apic", without "nointremap intel_iommu=off" the kexec loop test work well.
So it is x2apic, nmi_watchdog related...
Also do we know (like serial console log etc) how far ahead we > went in the kexec before we rebooted?
will add more printk after "Starting new kernel" to check it.
Thanks
Yinghai
| |