lkml.org 
[lkml]   [2011]   [Dec]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH 11/11] mm: Isolate pages for immediate reclaim on their own LRU
On Fri, Dec 16, 2011 at 04:07:28PM +0000, Mel Gorman wrote:
> On Fri, Dec 16, 2011 at 04:17:31PM +0100, Johannes Weiner wrote:
> > On Wed, Dec 14, 2011 at 03:41:33PM +0000, Mel Gorman wrote:
> > > It was observed that scan rates from direct reclaim during tests
> > > writing to both fast and slow storage were extraordinarily high. The
> > > problem was that while pages were being marked for immediate reclaim
> > > when writeback completed, the same pages were being encountered over
> > > and over again during LRU scanning.
> > >
> > > This patch isolates file-backed pages that are to be reclaimed when
> > > clean on their own LRU list.
> >
> > Excuse me if I sound like a broken record, but have those observations
> > of high scan rates persisted with the per-zone dirty limits patchset?
> >
>
> Unfortunately I wasn't testing that series. The focus of this series
> was primarily on THP-related stalls incurred by compaction which
> did not have a dependency on that series. Even with dirty balancing,
> similar stalls would be observed once dirty pages were in the zone
> at all.
>
> > In my tests with pzd, the scan rates went down considerably together
> > with the immediate reclaim / vmscan writes.
> >
>
> I probably should know but what is pzd?

Oops. Per-Zone Dirty limits.

> > Our dirty limits are pretty low - if reclaim keeps shuffling through
> > dirty pages, where are the 80% reclaimable pages?! To me, this sounds
> > like the unfair distribution of dirty pages among zones again. Is
> > there are a different explanation that I missed?
> >
>
> The alternative explanation is that the 20% dirty pages are all
> long-lived, at the end of the highest zone which is always scanned first
> so we continually have to scan over these dirty pages for prolonged
> periods of time.

That certainly makes sense to me and is consistent with your test case
having a fast producer of clean cache while the dirty cache is against
a slow backing device, so it may survive multiple full inactive cycles
before writeback finishes.

> > PS: It also seems a bit out of place in this series...?
>
> Without the last path, the System CPU time was stupidly high. In part,
> this is because we are no longer calling ->writepage from direct
> reclaim. If we were, the CPU usage would be far lower but it would
> be a lot slower too. It seemed remiss to leave system CPU usage that
> high without some explanation or patch dealing with it.
>
> The following replaces this patch with your series. dirtybalance-v7r1 is
> yours.
>
> 3.1.0-vanilla rc5-vanilla freemore-v6r1 isolate-v6r1 dirtybalance-v7r1
> System Time 1.22 ( 0.00%) 13.89 (-1040.72%) 46.40 (-3709.20%) 4.44 ( -264.37%) 43.05 (-3434.81%)
> +/- 0.06 ( 0.00%) 22.82 (-37635.56%) 3.84 (-6249.44%) 6.48 (-10618.92%) 4.04 (-6581.33%)
> User Time 0.06 ( 0.00%) 0.06 ( -6.90%) 0.05 ( 17.24%) 0.05 ( 13.79%) 0.05 ( 20.69%)
> +/- 0.01 ( 0.00%) 0.01 ( 33.33%) 0.01 ( 33.33%) 0.01 ( 39.14%) 0.01 ( -1.84%)
> Elapsed Time 10445.54 ( 0.00%) 2249.92 ( 78.46%) 70.06 ( 99.33%) 16.59 ( 99.84%) 73.71 ( 99.29%)
> +/- 643.98 ( 0.00%) 811.62 ( -26.03%) 10.02 ( 98.44%) 7.03 ( 98.91%) 17.90 ( 97.22%)
> THP Active 15.60 ( 0.00%) 35.20 ( 225.64%) 65.00 ( 416.67%) 70.80 ( 453.85%) 102.60 ( 657.69%)
> +/- 18.48 ( 0.00%) 51.29 ( 277.59%) 15.99 ( 86.52%) 37.91 ( 205.18%) 26.06 ( 141.02%)
> Fault Alloc 121.80 ( 0.00%) 76.60 ( 62.89%) 155.40 ( 127.59%) 181.20 ( 148.77%) 214.80 ( 176.35%)
> +/- 73.51 ( 0.00%) 61.11 ( 83.12%) 34.89 ( 47.46%) 31.88 ( 43.36%) 53.21 ( 72.39%)
> Fault Fallback 881.20 ( 0.00%) 926.60 ( -5.15%) 847.60 ( 3.81%) 822.00 ( 6.72%) 788.40 ( 10.53%)
> +/- 73.51 ( 0.00%) 61.26 ( 16.67%) 34.89 ( 52.54%) 31.65 ( 56.94%) 53.41 ( 27.35%)
> MMTests Statistics: duration
> User/Sys Time Running Test (seconds) 3540.88 1945.37 716.04 64.97 715.04
> Total Elapsed Time (seconds) 52417.33 11425.90 501.02 230.95 549.64
>
> Your series does help the System CPU time begining it from 46.4 seconds
> to 43.05 seconds. That is within the noise but towards the edge of
> one standard deviation. With such a small reduction, elapsed time was
> not helped. However, it did help THP allocation success rates - still
> within the noise but again at the edge of the noise which indicates
> a solid improvement.
>
> MMTests Statistics: vmstat
> Page Ins 3257266139 1111844061 17263623 10901575 20870385
> Page Outs 81054922 30364312 3626530 3657687 3665499
> Swap Ins 3294 2851 6560 4964 6598
> Swap Outs 390073 528094 620197 790912 604228
> Direct pages scanned 1077581700 3024951463 1764930052 115140570 1796314840
> Kswapd pages scanned 34826043 7112868 2131265 1686942 2093637
> Kswapd pages reclaimed 28950067 4911036 1246044 966475 1319662
> Direct pages reclaimed 805148398 280167837 3623473 2215044 4182274
> Kswapd efficiency 83% 69% 58% 57% 63%
> Kswapd velocity 664.399 622.521 4253.852 7304.360 3809.106
> Direct efficiency 74% 9% 0% 1% 0%
> Direct velocity 20557.737 264745.137 3522673.849 498551.938 3268166.145
> Percentage direct scans 96% 99% 99% 98% 99%
> Page writes by reclaim 722646 529174 620319 791018 604368
> Page writes file 332573 1080 122 106 140
> Page writes anon 390073 528094 620197 790912 604228
> Page reclaim immediate 0 2552514720 1635858848 111281140 1661416934
> Page rescued immediate 0 0 0 87848 0
> Slabs scanned 23552 23552 9216 8192 8192
> Direct inode steals 231 0 0 0 0
> Kswapd inode steals 0 0 0 0 0
> Kswapd skipped wait 28076 786 0 61 1
> THP fault alloc 609 383 753 906 1074
> THP collapse alloc 12 6 0 0 0
> THP splits 536 211 456 593 561
> THP fault fallback 4406 4633 4263 4110 3942
> THP collapse fail 120 127 0 0 0
> Compaction stalls 1810 728 623 779 869
> Compaction success 196 53 60 80 99
> Compaction failures 1614 675 563 699 770
> Compaction pages moved 193158 53545 243185 333457 409585
> Compaction move failure 9952 9396 16424 23676 30668
>
> The direct page scanned figure with your patch is still very high
> unfortunately.
>
> Overall, I would say that your series is not a replacement for the last
> patch in this series.

Agreed, thanks for clearing this up.


\
 
 \ /
  Last update: 2011-12-19 17:17    [W:0.220 / U:0.204 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site