Messages in this thread | | | Date | Mon, 24 Oct 2011 13:33:36 -0400 | From | Vivek Goyal <> | Subject | Re: kdump: crash_kexec()-smp_send_stop() race in panic |
| |
On Mon, Oct 24, 2011 at 10:07:19AM -0700, Eric W. Biederman wrote: > Américo Wang <xiyou.wangcong@gmail.com> writes: > > > On Mon, Oct 24, 2011 at 11:14 PM, Eric W. Biederman > > <ebiederm@xmission.com> wrote: > >> Michael Holzheu <holzheu@linux.vnet.ibm.com> writes: > >> > >>> Hello Vivek, > >>> > >>> In our tests we ran into the following scenario: > >>> > >>> Two CPUs have called panic at the same time. The first CPU called > >>> crash_kexec() and the second CPU called smp_send_stop() in panic() > >>> before crash_kexec() finished on the first CPU. So the second CPU > >>> stopped the first CPU and therefore kdump failed. > >>> > >>> 1st CPU: > >>> panic()->crash_kexec()->mutex_trylock(&kexec_mutex)-> do kdump > >>> > >>> 2nd CPU: > >>> panic()->crash_kexec()->kexec_mutex already held by 1st CPU > >>> ->smp_send_stop()-> stop CPU 1 (stop kdump) > >>> > >>> How should we fix this problem? One possibility could be to do > >>> smp_send_stop() before we call crash_kexec(). > >>> > >>> What do you think? > >> > >> smp_send_stop is insufficiently reliable to be used before crash_kexec. > >> > >> My first reaction would be to test oops_in_progress and wait until > >> oops_in_progress == 1 before calling smp_send_stop. > >> > > > > +1 > > > > One of my colleague mentioned the same problem with me inside > > RH, given the fact that the race condition window is small, it would > > not be easy to reproduce this scenario. > > As for reproducing it I have a hunch you could hack up something > horrible with smp_call_function and kprobes. > > > On a little more reflection we can't wait until oops_in_progress goes > to 1 before calling smp_send_stop. Because if crash_kexec is not > involved nothing we will never call smp_send_stop. > > So my second thought is to introduce another atomic variable > panic_in_progress, visible only in panic. The cpu that sets > increments panic_in_progress can call smp_send_stop. The rest of > the cpus can just go into a busy wait. That should stop nasty > fights about who is going to come out of smp_send_stop first.
Introducing panic_on_oops atomic sounds good.
Thanks Vivek -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |