Messages in this thread | | | Date | Fri, 16 Aug 2013 18:47:35 -0400 | From | Waiman Long <> | Subject | Re: [PATCH v3 0/3] qrwlock: Introducing a queue read/write lock implementation |
| |
On 08/14/2013 11:57 AM, Ingo Molnar wrote: > * Waiman Long<waiman.long@hp.com> wrote: > >> On 08/14/2013 06:20 AM, Ingo Molnar wrote: >>> * Waiman Long<waiman.long@hp.com> wrote: >>> >>>> I would like to share with you a rwlock related system crash that I >>>> encountered during my testing with hackbench on an 80-core DL980. The >>>> kernel crash because of a "watchdog detected hard lockup on cpu 79". The >>>> crashing CPU was running "write_lock_irq(&tasklist_lock)" in >>>> forget_original_parent() of the exit code path when I interrupted the >>>> hackbench which was spawning thousands of processes. Apparently, the >>>> remote CPU was not able to get the lock for a sufficient long time due >>>> to the unfairness of the rwlock which I think my version of queue rwlock >>>> will be able to alleviate this issue. >>>> >>>> So far, I was not able to reproduce the crash. I will try to see if I >>>> could more consistently reproduce it. >>> Was it an actual crash/lockup, or a longish hang followed by a lock >>> detector splat followed by the system eventually recovering back to >>> working order? >>> >>> Thanks, >>> >>> Ingo >> It was an actual crash initiated by the NMI handler. I think the >> system was in a halt state after that. > Could be a CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=1 kernel? > > Thanks, > > Ingo
My test system was a RHEL6.4 system. The 3.10 kernel config file was based on the original RHEL6.4 config file. So yes, the CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE parameter was set.
I also found that when I bump the process count up to about 30k range, interrupting the main hack_bench process may not cause the other spawned process to die out. I will further investigate this phenomenon later next week.
Regards, Longman
| |