lkml.org 
[lkml]   [2019]   [Jan]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: CFS scheduler: spin_lock usage causes dead lock when smp_apic_timer_interrupt occurs
On Fri, Jan 04, 2019 at 12:42:27PM +0000, Tom Putzeys wrote:
> Dear Ingo and Peter,
>
> I would like to report a possible bug in the CFS scheduler causing a
> dead lock. 
>
> We suspect this bug to have caused intermittent yet highly-persistent
> system freezes on our quad-core SMP systems.
>
> We noticed the problem on 4.1.17 preempt-rt but we suspect the
> problematic code is not linked to the preempt-rt patch and is also
> present in the latest 4.20 kernel.
>
> The problem concerns the use of spin_lock to lock cfs_b in a situation
> where the spin lock is used in an interrupt handler:

> - __run_hrtimer (in kernel/time/hrtimer.c) calls fn(timer) with IRQ's
> enabled. This can call sched_cfs_period_timer() (in
> kernel/sched/fair.c) which locks cfs_b. 

Hurmph, that's the softirq timer handling. And that is a nasty subtle
difference in context between softirq and hardirq timers.

Also, upstream doesn't use HRTIMER_MODE_SOFT here, but I suppose -rt
forces everything !HARD into SOFT.

> - the hard IRQ smp_apic_timer_interrupt can then occur. It can call
> ttwu_queue() which grabs the spin lock for its CPU run queue and can
> then try to enqueue a task via the CFS scheduler.

> - this can call check_enqueue_throttle() which can call
> assign_cfs_rq_runtime() which tries to obtain the cfs_b lock. It is
> now blocked.
>
> The cfs_b lock uses spin_lock and so was not intended for use inside a
> hard irq but the CFS scheduler does just that when it uses a
> hrtimer_interrupt to wake up and enqueue work. Our initial impression
> is that the cfs_b needs to be locked using spin_lock_irqsave.

I would expect lockdep you also complain about this, but yes, something
like that. I was very much expecting this to run with IRQs disabled (and
it does on mainline afaict).

\
 
 \ /
  Last update: 2019-01-07 11:32    [W:0.141 / U:0.208 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site