lkml.org 
[lkml]   [2017]   [Oct]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH v4 0/6] fs/dcache: Limit # of negative dentries
From
Date
On 09/18/2017 02:20 PM, Waiman Long wrote:
>
> A rogue application can potentially create a large number of negative
> dentries in the system consuming most of the memory available even if
> memory controller is enabled to limit memory usage. This can impact
> performance of other applications running on the system.
>
> We have customers seeing soft lockup and unresponsive system when
> tearing down a container because of the large number of negative
> dentries accumulated during its up time that had to be cleaned up at
> exit time when the container's filesystem was unmounted. So we need
> to do something about it.
>
> This patchset introduces changes to the dcache subsystem to limit
> the number of negative dentries allowed to be created thus limiting
> the amount of memory that can be consumed by negative dentries.
>
> Patch 1 just relocates the postion of the dentry_kill() function.
>
> Patch 2 tracks the number of negative dentries present in the LRU
> lists and reports it in /proc/sys/fs/dentry-state.
>
> Patch 3 sets a limit on the number of negative dentries allowable as a
> small percentage (2%) of total system memory. So the larger the system,
> the more negative dentries can be allowed. Once the limit is reached,
> new negative dentries will be killed after use.
>
> Patch 4 enables automatic pruning of least recently used negative
> dentries when it is close to the limit so that we won't end up killing
> recently used negative dentries.
>
> Patch 5 shows the number of forced negative dentry killings in
> /proc/sys/fs/dentry-state.
>
> Patch 6 enables auto-tuneup of free pool negative dentry count to
> no more than the maximum number of positive dentries ever used.
>
> With a 4.13 based kernel, the positive & negative dentries lookup rates
> (lookups per second) after initial boot on a 36-core 50GB memory VM
> with and without the patch were as follows:
>
> Metric w/o patch with patch
> ------ --------- ----------
> Positive dentry lookup 840269 845762
> Negative dentry lookup 1903405 1962514
> Negative dentry creation 6817957 6928768
>
> The last row refers to the creation rate of 1 millions negative
> dentries. With 50GB of memory, 1 millions negative dentries can be
> created with the patched kernel without any pruning or dentry killing.
>
> Ignoring some inherent noise in the test results, there wasn't any
> noticeable difference in term of lookup and negative dentry creation
> performance with or without this patch.
>
> By creating 10 millions negative dentries, however, the performance
> differed.
>
> Metric w/o patch with patch
> ------ --------- ----------
> Negative dentry creation 651663 190105
>
>
> For the patched kernel, the corresponding dentry-state was:
>
> 1608833 1590416 45 0 1579878 8286952
>
> This was expected as negative dentry creation throttling with forced
> dentry deletion happened in this case.
>
> Running the AIM7 high-systime workload on the same VM, the baseline
> performance was 186770 jobs/min. By running a single-thread rogue
> negative dentry creation program in the background until the patched
> kernel with 2% limit started throttling, the performance was 183746
> jobs/min. On an unpatched kernel with memory almost exhausted and
> memory shrinker was kicked in, the performance was 148997 jobs/min.
>
> So the patch does protect the system from suffering significant
> performance degradation in case a negative dentry creation rogue
> program is runninig in the background.
>
> Waiman Long (6):
> fs/dcache: Relocate dentry_kill() after lock_parent()
> fs/dcache: Track & report number of negative dentries
> fs/dcache: Limit numbers of negative dentries
> fs/dcache: Enable automatic pruning of negative dentries
> fs/dcache: Track count of negative dentries forcibly killed
> fs/dcache: Autotuning of negative dentry limit
>
> fs/dcache.c | 462 +++++++++++++++++++++++++++++++++++++++++++----
> include/linux/dcache.h | 8 +-
> include/linux/list_lru.h | 1 +
> mm/list_lru.c | 4 +-
> 4 files changed, 439 insertions(+), 36 deletions(-)
>
Ping!

Any comments on this v4 patchset. We have customers seeing problems
because of the unlimited growth of negative dentries. I am hoping that
at least part of this patchset, in some form, can be merged eventually.

Thanks,
Longman


\
 
 \ /
  Last update: 2017-10-05 17:42    [W:0.071 / U:1.956 seconds]
©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site