Messages in this thread | | | From | "Rafael J. Wysocki" <> | Subject | Re: [Query] Preemption (hogging) of the work handler | Date | Tue, 12 Jul 2016 14:24:47 +0200 |
| |
On Monday, July 11, 2016 03:46:01 PM Viresh Kumar wrote: > On 12-07-16, 00:44, Rafael J. Wysocki wrote: > > On Monday, July 11, 2016 03:35:01 PM Viresh Kumar wrote: > > > Hi Sergey and Jan, > > > > > > On 12-07-16, 00:44, Sergey Senozhatsky wrote: > > > > right. apart from cases when the existing console_unlock() behaviour can > > > > simply "block" a process to flush the log_buf to slow serial consoles > > > > (regardless the process execution context) and make the system less > > > > responsive, I have around ~10 absolutely different scenarios on my list that > > > > may cause soft/hard lockups, rcu stalls, oom-s, etc. and console_unlock() is > > > > the root cause there. the simplest ones involve heavy printk() usage, the > > > > trickier ones do not necessarily have anything that is abusing printk(): a > > > > moderate printk() pressure coming from other CPUs on the system and more or > > > > less active tty -> UART can do the trick, because uart interrupt service > > > > routine and call_console_drivers()->write() have to compete for the same > > > > uart port spin_lock. soft lockups are probably the most common problems, > > > > though, it's not all that easy to catch, because watchdog does not ring > > > > the bell straight after preempt_enable(), but from hrtimer interrupt, that > > > > happens approx every 4 seconds. by this time CPU can be somewhere far away > > > > from console_unlock(). I had an idea of doing watchdog soft lockup check > > > > from preempt_enable(), when it brings preempt_count down to zero, but not > > > > sure I can recall how well did it go. > > > > > > Thanks for your feedback guys, and I have one more blocking issue > > > where I need your help/advice. > > > > > > So, the excess printing in our case is done in parallel to system > > > suspend. And that can very much happen after all the non-boot CPUs are > > > offlined. > > > > > > Sometimes, the platform doesn't come back after suspend. I have tried > > > enabling no-console-suspend and the last line it prints is: > > > > > > Disabling non-boot CPUs > > > > > > And nothing after that at all. We have to forcefully reboot the phone > > > after that. Moving the prints to they synchronous way (using > > > echo 1 > /sys/module/printk/parameters/synchronous), fixes that issue. > > > > But no_console_suspend is best-effort by design. > > Yeah and I am not sure how should I go ahead about this issue now :)
FWIW, I think the reason why the "synchronous printk" works is because after disabling the non-boot CPU, the only remaining one disables local interrupts and won't do any async work any more until resume.
> > And *please* CC PM-related stuff to linux-pm. > > Sure. I wasn't sure initially when this thread got started, that it is > a PM related stuff and so didn't do it. As it was all about printk and > hogging :)
But you started to talk about suspend/resume and such at one point and that message should have been CCed to linux-pm.
And the reason why is because problems you see during suspend/resume may very well be suspend-specific and not visible otherwise. In which case you'll likely need input from the people on linux-pm.
Thanks, Rafael
| |