lkml.org 
[lkml]   [2020]   [Mar]   [11]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [patch] mm, oom: prevent soft lockup on memcg oom for UP systems
On Wed, 11 Mar 2020, Michal Hocko wrote:

> > > > When a process is oom killed as a result of memcg limits and the victim
> > > > is waiting to exit, nothing ends up actually yielding the processor back
> > > > to the victim on UP systems with preemption disabled. Instead, the
> > > > charging process simply loops in memcg reclaim and eventually soft
> > > > lockups.
> > > >
> > > > Memory cgroup out of memory: Killed process 808 (repro) total-vm:41944kB, anon-rss:35344kB, file-rss:504kB, shmem-rss:0kB, UID:0 pgtables:108kB oom_score_adj:0
> > > > watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [repro:806]
> > > > CPU: 0 PID: 806 Comm: repro Not tainted 5.6.0-rc5+ #136
> > > > RIP: 0010:shrink_lruvec+0x4e9/0xa40
> > > > ...
> > > > Call Trace:
> > > > shrink_node+0x40d/0x7d0
> > > > do_try_to_free_pages+0x13f/0x470
> > > > try_to_free_mem_cgroup_pages+0x16d/0x230
> > > > try_charge+0x247/0xac0
> > > > mem_cgroup_try_charge+0x10a/0x220
> > > > mem_cgroup_try_charge_delay+0x1e/0x40
> > > > handle_mm_fault+0xdf2/0x15f0
> > > > do_user_addr_fault+0x21f/0x420
> > > > page_fault+0x2f/0x40
> > > >
> > > > Make sure that something ends up actually yielding the processor back to
> > > > the victim to allow for memory freeing. Most appropriate place appears to
> > > > be shrink_node_memcgs() where the iteration of all decendant memcgs could
> > > > be particularly lengthy.
> > >
> > > There is a cond_resched in shrink_lruvec and another one in
> > > shrink_page_list. Why doesn't any of them hit? Is it because there are
> > > no pages on the LRU list? Because rss data suggests there should be
> > > enough pages to go that path. Or maybe it is shrink_slab path that takes
> > > too long?
> > >
> >
> > I think it can be a number of cases, most notably mem_cgroup_protected()
> > checks which is why the cond_resched() is added above it. Rather than add
> > cond_resched() only for MEMCG_PROT_MIN and for certain MEMCG_PROT_LOW, the
> > cond_resched() is added above the switch clause because the iteration
> > itself may be potentially very lengthy.
>
> Was any of the above the case for your soft lockup case? How have you
> managed to trigger it? As I've said I am not against the patch but I
> would really like to see an actual explanation what happened rather than
> speculations of what might have happened. If for nothing else then for
> the future reference.
>

Yes, this is how it was triggered in my own testing.

> If this is really about all the hierarchy being MEMCG_PROT_MIN protected
> and that results in a very expensive and pointless reclaim walk that can
> trigger soft lockup then it should be explicitly mentioned in the
> changelog.

I think the changelog clearly states that we need to guarantee that a
reclaimer will yield the processor back to allow a victim to exit. This
is where we make the guarantee. If it helps for the specific reason it
triggered in my testing, we could add:

"For example, mem_cgroup_protected() can prohibit reclaim and thus any
yielding in page reclaim would not address the issue."

\
 
 \ /
  Last update: 2020-03-11 20:46    [W:0.061 / U:3.364 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site