lkml.org 
[lkml]   [2017]   [Jul]   [17]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [rfc] superblock shrinker accumulating excessive deferred counts
On Mon, 17 Jul 2017, Dave Chinner wrote:

> > This is a side effect of super_cache_count() returning the appropriate
> > count but super_cache_scan() refusing to do anything about it and
> > immediately terminating with SHRINK_STOP, mostly for GFP_NOFS allocations.
>
> Yup. Happens during things like memory allocations in filesystem
> transaction context. e.g. when your memory pressure is generated by
> GFP_NOFS allocations within transactions whilst doing directory
> traversals (say 'chown -R' across an entire filesystem), then we
> can't do direct reclaim on the caches that are generating the memory
> pressure and so have to defer all the work to either kswapd or the
> next GFP_KERNEL allocation context that triggers reclaim.
>

Thanks for looking into this, Dave!

The number of GFP_NOFS allocations that build up the deferred counts can
be unbounded, however, so this can become excessive, and the oom killer
will not kill any processes in this context. Although the motivation to
do additional reclaim because of past GFP_NOFS reclaim attempts is
worthwhile, I think it should be limited because currently it only
increases until something is able to start draining these excess counts.
Having 10,000 GFP_NOFS reclaim attempts store up
(2 * nr_scanned * freeable) / (nr_eligible + 1) objects 10,000 times
such that it exceeds freeable by many magnitudes doesn't seem like a
particularly useful thing. For reference, we have seen nr_deferred for a
single node to be > 10,000,000,000 in practice. total_scan is limited to
2 * freeable for each call to do_shrink_slab(), but such an excessive
deferred count will guarantee it retries 2 * freeable each time instead of
the proportion of lru scanned as intended.

What breaks if we limit the nr_deferred counts to freeable * 4, for
example?

> > and no matter how much __GFP_FS scanning is done
> > capped by total_scan, we can never fully get down to batch_count == 1024.
>
> I don't see a batch_count variable in the shrinker code anywhere,
> so I'm not sure what you mean by this.
>

batch_size == 1024, sorry.

> Can you post a shrinker trace that shows the deferred count wind
> up and then display the problem you're trying to describe?
>

All threads contending on the list_lru's nlru->lock because they are all
stuck in super_cache_count() while one thread is iterating through an
excessive number of deferred objects in super_cache_scan(), contending for
the same locks and nr_deferred never substantially goes down.

The problem with the superblock shrinker, which is why I emailed Al
originally, is also that it is SHRINKER_MEMCG_AWARE. Our
list_lru_shrink_count() is only representative for the list_lru of
sc->memcg, which is used in both super_cache_count() and
super_cache_scan() for various math. The nr_deferred counts from the
do_shrink_slab() logic, however, are per-nid and, as such, various memcgs
get penalized with excessive counts that they do not have freeable to
begin with.

\
 
 \ /
  Last update: 2017-07-17 22:37    [W:0.100 / U:0.288 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site