lkml.org 
[lkml]   [2016]   [Nov]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`
    From
    Date
    On 24.11.2016 11:15, Michal Hocko wrote:

    > On Mon 21-11-16 16:35:53, Donald Buczek wrote:
    > [...]
    >> Hello,
    >>
    >> thanks a lot for looking into this!
    >>
    >> Let me add some information from the reporting site:
    >>
    >> * We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov
    >> 2016) and it doesn't shut up the rcu stall warnings.
    >>
    >> * Log file from a boot with the patch applied ( grep kernel
    >> /var/log/messages ) is here :
    >> http://owww.molgen.mpg.de/~buczek/321322/2016-11-21_syslog.txt
    >>
    >> * This system is a backup server and walks over thousands of files sometimes
    >> with multiple parallel rsync processes.
    >>
    >> * No rcu_* warnings on that machine with 4.7.2, but with 4.8.4 , 4.8.6 ,
    >> 4.8.8 and now 4.9.0-rc5+Pauls patch
    > I assume you haven't tried the Linus 4.8 kernel without any further
    > stable patches? Just to be sure we are not talking about some later
    > regression which found its way to the stable tree.
    >
    >> * When the backups are actually happening there might be relevant memory
    >> pressure from inode cache and the rsync processes. We saw the oom-killer
    >> kick in on another machine with same hardware and similar (a bit higher)
    >> workload. This other machine also shows a lot of rcu stall warnings since
    >> 4.8.4.
    >>
    >> * We see "rcu_sched detected stalls" also on some other machines since we
    >> switched to 4.8 but not as frequently as on the two backup servers. Usually
    >> there's "shrink_node" and "kswapd" on the top of the stack. Often
    >> "xfs_reclaim_inodes" variants on top of that.
    > I would be interested to see some reclaim tracepoints enabled. Could you
    > try that out? At least mm_shrink_slab_{start,end} and
    > mm_vmscan_lru_shrink_inactive. This should tell us more about how the
    > reclaim behaved.

    We'll try that tomorrow!

    Donald

    --
    Donald Buczek
    buczek@molgen.mpg.de
    Tel: +49 30 8413 1433

    \
     
     \ /
      Last update: 2016-11-24 19:51    [W:4.074 / U:0.100 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site