lkml.org 
[lkml]   [2011]   [May]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC][PATCH v3 0/10] memcg async reclaim
    On Thu, 26 May 2011 18:49:26 -0700
    Ying Han <yinghan@google.com> wrote:

    > On Wed, May 25, 2011 at 10:10 PM, KAMEZAWA Hiroyuki
    > <kamezawa.hiroyu@jp.fujitsu.com> wrote:
    > >
    > > It's now merge window...I just dump my patch queue to hear other's idea.
    > > I wonder I should wait until dirty_ratio for memcg is queued to mmotm...
    > > I'll be busy with LinuxCon Japan etc...in the next week.
    > >
    > > This patch is onto mmotm-May-11 + some patches queued in mmotm, as numa_stat.
    > >
    > > This is a patch for memcg to keep margin to the limit in background.
    > > By keeping some margin to the limit in background, application can
    > > avoid foreground memory reclaim at charge() and this will help latency.
    > >
    > > Main changes from v2 is.
    > >  - use SCHED_IDLE.
    > >  - removed most of heuristic codes. Now, code is very simple.
    > >
    > > By using SCHED_IDLE, async memory reclaim can only consume 0.3%? of cpu
    > > if the system is truely busy but can use much CPU if the cpu is idle.
    > > Because my purpose is for reducing latency without affecting other running
    > > applications, SCHED_IDLE fits this work.
    > >
    > > If application need to stop by some I/O or event, background memory reclaim
    > > will cull memory while the system is idle.
    > >
    > > Perforemce:
    > >  Running an httpd (apache) under 300M limit. And access 600MB working set
    > >  with normalized distribution access by apatch-bench.
    > >  apatch bench's concurrency was 4 and did 40960 accesses.
    > >
    > > Without async reclaim:
    > > Connection Times (ms)
    > >              min  mean[+/-sd] median   max
    > > Connect:        0    0   0.0      0       2
    > > Processing:    30   37  28.3     32    1793
    > > Waiting:       28   35  25.5     31    1792
    > > Total:         30   37  28.4     32    1793
    > >
    > > Percentage of the requests served within a certain time (ms)
    > >  50%     32
    > >  66%     32
    > >  75%     33
    > >  80%     34
    > >  90%     39
    > >  95%     60
    > >  98%    100
    > >  99%    133
    > >  100%   1793 (longest request)
    > >
    > > With async reclaim:
    > > Connection Times (ms)
    > >              min  mean[+/-sd] median   max
    > > Connect:        0    0   0.0      0       2
    > > Processing:    30   35  12.3     32     678
    > > Waiting:       28   34  12.0     31     658
    > > Total:         30   35  12.3     32     678
    > >
    > > Percentage of the requests served within a certain time (ms)
    > >  50%     32
    > >  66%     32
    > >  75%     33
    > >  80%     34
    > >  90%     39
    > >  95%     49
    > >  98%     71
    > >  99%     86
    > >  100%    678 (longest request)
    > >
    > >
    > > It seems latency is stabilized by hiding memory reclaim.
    > >
    > > The score for memory reclaim was following.
    > > See patch 10 for meaning of each member.
    > >
    > > == without async reclaim ==
    > > recent_scan_success_ratio 44
    > > limit_scan_pages 388463
    > > limit_freed_pages 162238
    > > limit_elapsed_ns 13852159231
    > > soft_scan_pages 0
    > > soft_freed_pages 0
    > > soft_elapsed_ns 0
    > > margin_scan_pages 0
    > > margin_freed_pages 0
    > > margin_elapsed_ns 0
    > >
    > > == with async reclaim ==
    > > recent_scan_success_ratio 6
    > > limit_scan_pages 0
    > > limit_freed_pages 0
    > > limit_elapsed_ns 0
    > > soft_scan_pages 0
    > > soft_freed_pages 0
    > > soft_elapsed_ns 0
    > > margin_scan_pages 1295556
    > > margin_freed_pages 122450
    > > margin_elapsed_ns 644881521
    > >
    > >
    > > For this case, SCHED_IDLE workqueue can reclaim enough memory to the httpd.
    > >
    > > I may need to dig why scan_success_ratio is far different in the both case.
    > > I guess the difference of epalsed_ns is because several threads enter
    > > memory reclaim when async reclaim doesn't run. But may not...
    > >
    >
    >
    > Hmm.. I noticed a very strange behavior on a simple test w/ the patch set.
    >
    > Test:
    > I created a 4g memcg and start doing cat. Then the memcg being OOM
    > killed as soon as it reaches its hard_limit. We shouldn't hit OOM even
    > w/o async-reclaim.
    >
    > Again, I will read through the patch. But like to post the test result first.
    >
    > $ echo $$ >/dev/cgroup/memory/A/tasks
    > $ cat /dev/cgroup/memory/A/memory.limit_in_bytes
    > 4294967296
    >
    > $ time cat /export/hdc3/dd_A/tf0 > /dev/zero
    > Killed
    >

    I did the same kind of test without any problem...but ok, I'll do more test
    later.



    > real 0m53.565s
    > user 0m0.061s
    > sys 0m4.814s
    >
    > Here is the OOM log:
    >
    > May 26 18:43:00 kernel: [ 963.489112] cat invoked oom-killer:
    > gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
    > May 26 18:43:00 kernel: [ 963.489121] Pid: 9425, comm: cat Tainted:
    > G W 2.6.39-mcg-DEV #131
    > May 26 18:43:00 kernel: [ 963.489123] Call Trace:
    > May 26 18:43:00 kernel: [ 963.489134] [<ffffffff810e3512>]
    > dump_header+0x82/0x1af
    > May 26 18:43:00 kernel: [ 963.489137] [<ffffffff810e33ca>] ?
    > spin_lock+0xe/0x10
    > May 26 18:43:00 kernel: [ 963.489140] [<ffffffff810e33f9>] ?
    > find_lock_task_mm+0x2d/0x67
    > May 26 18:43:00 kernel: [ 963.489143] [<ffffffff810e38dd>]
    > oom_kill_process+0x50/0x27b
    > May 26 18:43:00 kernel: [ 963.489155] [<ffffffff810e3dc6>]
    > mem_cgroup_out_of_memory+0x9a/0xe4
    > May 26 18:43:00 kernel: [ 963.489160] [<ffffffff811153aa>]
    > mem_cgroup_handle_oom+0x134/0x1fe
    > May 26 18:43:00 kernel: [ 963.489163] [<ffffffff81114a72>] ?
    > __mem_cgroup_insert_exceeded+0x83/0x83
    > May 26 18:43:00 kernel: [ 963.489176] [<ffffffff811166e9>]
    > __mem_cgroup_try_charge.clone.3+0x368/0x43a
    > May 26 18:43:00 kernel: [ 963.489179] [<ffffffff81117586>]
    > mem_cgroup_cache_charge+0x95/0x123
    > May 26 18:43:00 kernel: [ 963.489183] [<ffffffff810e16d8>]
    > add_to_page_cache_locked+0x42/0x114
    > May 26 18:43:00 kernel: [ 963.489185] [<ffffffff810e17db>]
    > add_to_page_cache_lru+0x31/0x5f
    > May 26 18:43:00 kernel: [ 963.489189] [<ffffffff81145636>]
    > mpage_readpages+0xb6/0x132
    > May 26 18:43:00 kernel: [ 963.489194] [<ffffffff8119992f>] ?
    > noalloc_get_block_write+0x24/0x24
    > May 26 18:43:00 kernel: [ 963.489197] [<ffffffff8119992f>] ?
    > noalloc_get_block_write+0x24/0x24
    > May 26 18:43:00 kernel: [ 963.489201] [<ffffffff81036742>] ?
    > __switch_to+0x160/0x212
    > May 26 18:43:00 kernel: [ 963.489205] [<ffffffff811978b2>]
    > ext4_readpages+0x1d/0x1f
    > May 26 18:43:00 kernel: [ 963.489209] [<ffffffff810e8d4b>]
    > __do_page_cache_readahead+0x144/0x1e3
    > May 26 18:43:00 kernel: [ 963.489212] [<ffffffff810e8e0b>]
    > ra_submit+0x21/0x25
    > May 26 18:43:00 kernel: [ 963.489215] [<ffffffff810e9075>]
    > ondemand_readahead+0x18c/0x19f
    > May 26 18:43:00 kernel: [ 963.489218] [<ffffffff810e9105>]
    > page_cache_async_readahead+0x7d/0x86
    > May 26 18:43:00 kernel: [ 963.489221] [<ffffffff810e2b7e>]
    > generic_file_aio_read+0x2d8/0x5fe
    > May 26 18:43:00 kernel: [ 963.489225] [<ffffffff81119626>]
    > do_sync_read+0xcb/0x108
    > May 26 18:43:00 kernel: [ 963.489230] [<ffffffff811f168a>] ?
    > fsnotify_perm+0x66/0x72
    > May 26 18:43:00 kernel: [ 963.489233] [<ffffffff811f16f7>] ?
    > security_file_permission+0x2e/0x33
    > May 26 18:43:00 kernel: [ 963.489236] [<ffffffff8111a0c8>]
    > vfs_read+0xab/0x107
    > May 26 18:43:00 kernel: [ 963.489239] [<ffffffff8111a1e4>] sys_read+0x4a/0x6e
    > May 26 18:43:00 kernel: [ 963.489244] [<ffffffff8140f469>]
    > sysenter_dispatch+0x7/0x27
    > May 26 18:43:00 kernel: [ 963.489248] Task in /A killed as a result
    > of limit of /A
    > May 26 18:43:00 kernel: [ 963.489251] memory: usage 4194304kB, limit
    > 4194304kB, failcnt 26
    > May 26 18:43:00 kernel: [ 963.489253] memory+swap: usage 0kB, limit
    > 9007199254740991kB, failcnt 0
    >

    Hmm, why memory+swap usage 0kb here...

    In this set, I used mem_cgroup_margin() rather than res_counter_margin().
    Hmm, do you disable swap accounting ? If so, I may miss some.

    Thanks,
    -Kame




    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2011-05-27 04:27    [W:0.037 / U:63.104 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site