lkml.org 
[lkml]   [2016]   [Jan]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code
    From
    Date
    On 01/28/2016 09:28 PM, Sergey Senozhatsky wrote:
    > On (01/28/16 20:32), Peter Hurley wrote:
    > [..]
    >> You're assuming that Byungchul's patch is relevant to the recursion
    >> he witnessed. There are several paths into spin_dump().
    >
    > yes. I was speaking in the context of Byungchul's report.
    >
    >> Here's one that doesn't wait at all:
    >>
    >> vprintk_emit()
    >> console_trylock()
    >> down_trylock()
    >> raw_spin_lock_irqsave()
    >> ...
    >> do_raw_spin_lock()
    >> debug_spin_lock_before()
    >> SPIN_BUG_ON()
    >> spin_bug()
    >> spin_dump()
    >> printk()
    >> ** RINSE AND REPEAT **
    >
    > ah, yes, agree.
    >
    >>>> Additionally, what if the console_sem is simply corrupted?
    >>>> A livelock with no output ever is not very helpful.
    >>>
    >>> if it's corrupted then this is not a spinlock debug problem.
    >>> at all.
    >>
    >> I don't follow you.
    >>
    >
    > indeed very misleading, sorry, almost didn't sleep last nigh.
    > removing SPIN_BUG_ON entirely is not my logic, not all. printk locks are
    > special, I agree. in context of the proposed patch - we can't disable
    > spin_dump() for printk locks if they were corrupted. for printk locks it's
    > over, nothing will be printed anymore. the only thing that _may be_ will
    > help is zap_locks(), but not 100% guaranteed... we can panic the system,
    > probably, if printk locks are getting corrupted, but panic() will not do the
    > trick in general case, at this point -- console_unlock() takes the logbuf_lock,
    > which can be corrupted. apart from that console driver can be in a weird state.
    >
    > I sort of proposed to update console_flush_on_panic() (called from panic())
    > function a while ago to do zap_locks(), so in this case declaring BUG() from
    > spinlock debug when we see 'bad' printk-related locks will have better
    > chances to work out (assuming that console driver(-s) is (are) not against
    > us).

    Yeah, exactly, something that improves the chances of successful output.


    > [..]
    >> This was in reference to a problem with spin lock recursion that
    >> can't print. The spin lock recursion deadlocks, but you'll never
    >> see the diagnostic because the driver is already holding the lock
    >> (not from printk() but from some other code).
    >>
    >> The printk doesn't even need to be directly related to the
    >> console driver itself, but some other thing that the console driver
    >> depends on while holding the spin lock that it claims for console output.
    >
    > aha, ok. yes, this is certainly possible.
    >
    >>> this is not a case of printk recursion and it should be handled
    >>> just fine. console drivers are called under console_sem only.
    >>> logbuf lock is unlocked. vprintk_emit() adds message to the logbuf,
    >>> calls console_trylock() (which of course does not lock anything)
    >>> and returns back to console_driver code.
    >>
    >> Not if locks are zapped because printk() recognizes a recursion.
    >> Note console driver's locks are not zapped. For example,
    >
    > yes, I proposed to add a ->reset callback to struct console
    > a while ago, and to do a console reset loop in zap_locks()

    What was the patch series title? I'd like to review that.

    That would solve the recursive deadlock from console driver as well
    (at least with CONFIG_DEBUG_SPINLOCK) because the printk() recursion
    would zap the locks including the console driver's lock and
    at least get the last output so that we'd know there was a recursion,
    and fix it.


    > zap_locks:
    > ...
    > for_each_console(con)
    > if (con->reset)
    > con->reset(con)
    >
    > that would re-init console drivers (locks, etc.).
    >
    >
    > IOW, panic() does zap_locks(), zap_locks() zap the locks and
    > resets the console drivers. that's sort of what I have in my
    > private kernels.
    >
    > [..]
    >>> the only case when we really have a printk recursion is when
    >>> someone calls printk() from within the vprintk_emit() logbuf_lock
    >>> area.
    >>
    >> Not true.
    >>
    >> A while back, Jan Kara reworked the call site around
    >> console_trylock_from_printk(). Hung with no output under unknown
    >> conditions [1].
    >>
    >> Never solved, but obviously means there are unhandled recursions.

    I'd still like to enable lockdep for console drivers, but I need a
    better plan to debug unknown printk() recursions.


    > aha, ok.
    >
    > -ss
    >

    \
     
     \ /
      Last update: 2016-01-29 07:21    [W:3.558 / U:0.108 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site