lkml.org 
[lkml]   [2018]   [May]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [rfc patch] mm, oom: fix unnecessary killing of additional processes
On Thu, 31 May 2018, Michal Hocko wrote:

> > It's not a random timeout, it's sufficiently long such that we don't oom
> > kill several processes needlessly in the very rare case where oom livelock
> > would actually prevent the original victim from exiting. The oom reaper
> > processing an mm, finding everything to be mlocked, and immediately
> > MMF_OOM_SKIP is inappropriate. This is rather trivial to reproduce for a
> > large memory hogging process that mlocks all of its memory; we
> > consistently see spurious and unnecessary oom kills simply because the oom
> > reaper has set MMF_OOM_SKIP very early.
>
> It takes quite some additional steps for admin to allow a large amount
> of mlocked memory and such an application should be really careful to
> not consume too much memory. So how come this is something you see that
> consistently? Is this some sort of bug or an unfortunate workload side
> effect? I am asking this because I really want to see how relevant this
> really is.
>

The bug is that the oom reaper sets MMF_OOM_SKIP almost immediately after
the victim has been chosen for oom kill and we get follow-up oom kills,
not that the process is able to mlock a large amount of memory. Mlock
here is only being discussed as a single example. Tetsuo has brought up
the example of all shared file-backed memory. We've discussed the mm
having a single blockable mmu notifier. Regardless of how we arrive at
the point where the oom reaper can't free memory, which could be any of
those three cases, if (1) the original victim is sufficiently large that
follow-up oom kills would become unnecessary and (2) other threads
allocate/charge before the oom victim reaches exit_mmap(), this occurs.

We have examples of cases where oom reaping was successful, but the rss
numbers in the kernel log are very similar to when it was oom killed and
the process is known not to mlock, the reason is because the oom reaper
could free very little memory due to blockable mmu notifiers.

> But the waiting periods just turn out to be a really poor design. There
> will be no good timeout to fit for everybody. We can do better and as
> long as this is the case the timeout based solution should be really
> rejected. It is a shortcut that doesn't really solve the underlying
> problem.
>

The current implementation is a timeout based solution for mmap_sem, it
just has the oom reaper spinning trying to grab the sem and eventually
gives up. This patch allows it to currently work on other mm's and
detects the timeout in a different way, with jiffies instead of an
iterator.

I'd love a solution where we can reliably detect an oom livelock and oom
kill additional processes but only after the original victim has had a
chance to do exit_mmap() without a timeout, but I don't see one being
offered. Given Tetsuo has seen issues with this in the past and suggested
a similar proposal means we are not the only ones feeling pain from this.

> > I'm open to hearing any other suggestions that you have other than waiting
> > some time period before MMF_OOM_SKIP gets set to solve this problem.
>
> I've already offered one. Make mlocked pages reapable.

Making mlocked pages reapable would only solve the most trivial reproducer
of this. Unless the oom reaper can guarantee that it will never block and
can free all memory that exit_mmap() can free, we need to ensure that a
victim has a chance to reach the exit path on its own before killing every
other process on the system.

I'll fix the issue I identified with doing list_add_tail() rather than
list_add(), fix up the commit message per Tetsuo to identify the other
possible ways this can occur other than mlock, remove the rfc tag, and
repost.

\
 
 \ /
  Last update: 2018-05-31 23:17    [W:0.638 / U:0.220 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site