lkml.org 
[lkml]   [2015]   [Jan]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: RCU CPU stall console spews leads to soft lockup disabled is reasonable ?
    On 2015/1/19 22:06, Don Zickus wrote:
    > On Mon, Jan 19, 2015 at 05:04:29PM +0800, Zhang Zhen wrote:
    >>>
    >>> Did you really intend to acquire the same spinlock twice in a row,
    >>> forcing a self-deadlock? If not, I of course suggest changing the second
    >>> "spin_lock()" to "spin_unlock()".
    >>>
    >>
    >> Yes, i acquire the same spinlock twice in order to reproduce the problem.
    >>
    >>> If your .config has CONFIG_TREE_RCU=y, the above is quite likely to
    >>> give you an RCU CPU stall warning.
    >>>
    >>
    >> In my .config CONFIG_TREE_RCU=y.
    >>
    >> If softlockup_thresh < rcu_cpu_stall_timeout, it will give soft lockup warning.
    >> If softlockup_thresh > rcu_cpu_stall_timeout, it will likely to give RCU CPU stall warning
    >> just like above and no give soft lockup warning.
    >>
    >> It means that RCU CPU stall console spews leads to soft lockup disabled.
    >> Is this reasonable ?
    >
    > I believe so. In kernel v3.10.., all activity to the console executed
    > touch_nmi_watchdog() which calls touch_softlockup_watchdog, which delayed
    > the softlockup for another round of 'softlockup_thresh'.
    >
    Yeah, you are right. It's the real reason.

    > Of course back then, touch_nmi_watchdog touched all cpus. So a problem
    > like this was masked. I believe this upstream commit 62572e29bc53, solved
    > the problem.

    Thanks for your suggestion.

    Commit 62572e29bc53 changed the semantics of touch_nmi_watchdog and make it
    only touch local cpu not every one.
    But watchdog_nmi_touch = true only guarantee no hard lockup check on this cpu.

    Commit 62572e29bc53 didn't changed the semantics of touch_softlockup_watchdog.
    >
    > You can apply that commit and see if you if you get both RCU stall
    > messages _and_ softlockup messages. I believe that is what you were
    > expecting, correct?
    >
    Correct, i expect i can get both RCU stall messages _and_ softlockup messages.
    I applied that commit, and i only got RCU stall messages.

    / #
    / # echo 60 > /proc/sys/kernel/watchdog_thresh
    / # busybox insmod softlockup_test.ko
    [ 35.344060] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=21002 jiffies, g=85, c=84, q=4)
    [ 35.344060] INFO: Stall ended before state dump start
    [ 98.349079] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=84007 jiffies, g=85, c=84, q=4)
    [ 98.349079] INFO: Stall ended before state dump start
    [ 161.354100] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=147012 jiffies, g=85, c=84, q=4)
    [ 161.354100] INFO: Stall ended before state dump start
    >
    > Of course, on a non-virt guest, your test case would normally trigger a
    > hardlockup warning first. And a later kernel version for the guest may
    > actually do that (not quite sure if the emulated PMU stuff is upstream or
    > not yet). Just to set your expectations correctly.
    >
    Yes, on a non-virt guest, my test case tiggered hardlockup warning firt.

    Best regards!
    > Cheers,
    > Don
    >
    >>
    >> Thanks!
    >>
    >>>> return 0;
    >>>> }
    >>>>
    >>>> static int __init test_init(void)
    >>>> {
    >>>> hello_start();
    >>>>
    >>>> printk(KERN_INFO "Module init\n");
    >>>> return 0;
    >>>> }
    >>>>
    >>>> static void __exit test_exit(void)
    >>>> {
    >>>> printk(KERN_INFO "Module exit!\n");
    >>>> }
    >>>>
    >>>> module_init(test_init)
    >>>> module_exit(test_exit)
    >>>> MODULE_LICENSE("GPL");
    >>>> //
    >>>>
    >>>> My kernel version is v3.10.63, and i checked the kernel source code,
    >>>>
    >>>> update_process_times
    >>>> -> run_local_timers
    >>>> -> hrtimer_run_queues
    >>>> -> __run_hrtimer
    >>>> -> watchdog_timer_fn
    >>>> -> is_softlockup
    >>>>
    >>>> -> rcu_check_callbacks
    >>>> -> __rcu_pending
    >>>> -> check_cpu_stall
    >>>> -> print_cpu_stall
    >>>>
    >>>> If softlockup_thresh > rcu_cpu_stall_timeout, print_cpu_stall will print log to serial port.
    >>>>
    >>>> The 8250 serial driver will call serial8250_console_write => touch_nmi_watchdog() which reset
    >>>> watchdog_touch_ts to 0. So the softlockup will not be triggered.
    >>>>
    >>>> Is this reasonable? Why?
    >>>
    >>> Is exactly what reasonable? ;-)
    >>>
    >>> Yes, it is reasonable that your code triggers an RCU CPU stall warning.
    >>>
    >>> No, it is not reasonable that the RCU CPU stall warning does not include
    >>> a stack trace, and the fix for that bug will be going into the next merge
    >>> window.
    >>>
    >>> Yes, is is reasonable that varying the softlockup and RCU CPU stall
    >>> timeouts might change the behavior.
    >>>
    >>> No, your code is not reasonable, except perhaps as a test of the
    >>> generation of softlockup and RCU CPU stall warnings. If you are not
    >>> trying to test softlockup and RCU CPU stall warnings, you should of course
    >>> not try to acquire any non-recursive exclusive lock that you already hold.
    >>>
    >>>> If it is not reasonable, we should adjust the printk loglevel from *KERN_ERR* to *KERN_INFO*
    >>>> in print_cpu_stall.
    >>>
    >>> Given that RCU CPU stall warnings are supposed to be pointing out errors
    >>> elsewhere in the kernel, and in this case are pointing out errors elsewhere
    >>> in the kernel, namely in your hello_start() function, it is reasonable
    >>> that the RCU CPU stall warnings use the KERN_ERR loglevel.
    >>>
    >>> Or am I missing something here?
    >>>
    >>> Thanx, Paul
    >>>
    >>>
    >>> .
    >>>
    >>
    >>
    >
    > .
    >




    \
     
     \ /
      Last update: 2015-01-20 04:21    [W:4.119 / U:0.160 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site