lkml.org 
[lkml]   [2017]   [Sep]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [patch 10/29] lockup_detector/perf: Prevent cpu hotplug deadlock
On Fri, Sep 01, 2017 at 09:29:07PM +0200, Thomas Gleixner wrote:
> On Fri, 1 Sep 2017, Don Zickus wrote:
> > On Thu, Aug 31, 2017 at 09:16:08AM +0200, Thomas Gleixner wrote:
> > > The following deadlock is possible in the watchdog hotplug code:
> > >
> > > cpus_write_lock()
> > > ...
> > > takedown_cpu()
> > > smpboot_park_threads()
> > > smpboot_park_thread()
> > > kthread_park()
> > > ->park() := watchdog_disable()
> > > watchdog_nmi_disable()
> > > perf_event_release_kernel();
> > > put_event()
> > > _free_event()
> > > ->destroy() := hw_perf_event_destroy()
> > > x86_release_hardware()
> > > release_ds_buffers()
> > > get_online_cpus()
> > >
> > > when a per cpu watchdog perf event is destroyed which drops the last
> > > reference to the PMU hardware. The cleanup code there invokes
> > > get_online_cpus() which instantly deadlocks because the hotplug percpu
> > > rwsem is write locked.
> >
> > The main reason perf_event_release_kernel is in this path is because the
> > oprofile folks complained they couldn't use the perf counters when the
> > nmi_watchdog was disabled on the command line.
>
> If the nmi watchdog is disabled on the command line then there are no
> counters claimed at all.

Ah, I see it now. When you park all the threads, you clear the cpumask
which then calls the lockup detector_cleanup to release all perf_counters
that are no longer used by the cpumask.

Further reading this code, I see how the code covers various race conditions
we tried solving in the past.

Thanks!

Cheers,
Don

\
 
 \ /
  Last update: 2017-09-05 16:51    [W:0.077 / U:0.076 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site