Messages in this thread | | | Subject | Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code | From | Peter Hurley <> | Date | Thu, 28 Jan 2016 21:48:49 -0800 |
| |
On 01/28/2016 09:28 PM, Sergey Senozhatsky wrote: > On (01/28/16 20:32), Peter Hurley wrote: > [..] >> You're assuming that Byungchul's patch is relevant to the recursion >> he witnessed. There are several paths into spin_dump(). > > yes. I was speaking in the context of Byungchul's report. > >> Here's one that doesn't wait at all: >> >> vprintk_emit() >> console_trylock() >> down_trylock() >> raw_spin_lock_irqsave() >> ... >> do_raw_spin_lock() >> debug_spin_lock_before() >> SPIN_BUG_ON() >> spin_bug() >> spin_dump() >> printk() >> ** RINSE AND REPEAT ** > > ah, yes, agree. > >>>> Additionally, what if the console_sem is simply corrupted? >>>> A livelock with no output ever is not very helpful. >>> >>> if it's corrupted then this is not a spinlock debug problem. >>> at all. >> >> I don't follow you. >> > > indeed very misleading, sorry, almost didn't sleep last nigh. > removing SPIN_BUG_ON entirely is not my logic, not all. printk locks are > special, I agree. in context of the proposed patch - we can't disable > spin_dump() for printk locks if they were corrupted. for printk locks it's > over, nothing will be printed anymore. the only thing that _may be_ will > help is zap_locks(), but not 100% guaranteed... we can panic the system, > probably, if printk locks are getting corrupted, but panic() will not do the > trick in general case, at this point -- console_unlock() takes the logbuf_lock, > which can be corrupted. apart from that console driver can be in a weird state. > > I sort of proposed to update console_flush_on_panic() (called from panic()) > function a while ago to do zap_locks(), so in this case declaring BUG() from > spinlock debug when we see 'bad' printk-related locks will have better > chances to work out (assuming that console driver(-s) is (are) not against > us).
Yeah, exactly, something that improves the chances of successful output.
> [..] >> This was in reference to a problem with spin lock recursion that >> can't print. The spin lock recursion deadlocks, but you'll never >> see the diagnostic because the driver is already holding the lock >> (not from printk() but from some other code). >> >> The printk doesn't even need to be directly related to the >> console driver itself, but some other thing that the console driver >> depends on while holding the spin lock that it claims for console output. > > aha, ok. yes, this is certainly possible. > >>> this is not a case of printk recursion and it should be handled >>> just fine. console drivers are called under console_sem only. >>> logbuf lock is unlocked. vprintk_emit() adds message to the logbuf, >>> calls console_trylock() (which of course does not lock anything) >>> and returns back to console_driver code. >> >> Not if locks are zapped because printk() recognizes a recursion. >> Note console driver's locks are not zapped. For example, > > yes, I proposed to add a ->reset callback to struct console > a while ago, and to do a console reset loop in zap_locks()
What was the patch series title? I'd like to review that.
That would solve the recursive deadlock from console driver as well (at least with CONFIG_DEBUG_SPINLOCK) because the printk() recursion would zap the locks including the console driver's lock and at least get the last output so that we'd know there was a recursion, and fix it.
> zap_locks: > ... > for_each_console(con) > if (con->reset) > con->reset(con) > > that would re-init console drivers (locks, etc.). > > > IOW, panic() does zap_locks(), zap_locks() zap the locks and > resets the console drivers. that's sort of what I have in my > private kernels. > > [..] >>> the only case when we really have a printk recursion is when >>> someone calls printk() from within the vprintk_emit() logbuf_lock >>> area. >> >> Not true. >> >> A while back, Jan Kara reworked the call site around >> console_trylock_from_printk(). Hung with no output under unknown >> conditions [1]. >> >> Never solved, but obviously means there are unhandled recursions.
I'd still like to enable lockdep for console drivers, but I need a better plan to debug unknown printk() recursions.
> aha, ok. > > -ss >
| |