lkml.org 
[lkml]   [2013]   [Jan]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH for 3.2.34] memcg: do not trigger OOM from add_to_page_cache_locked
On Fri 25-01-13 16:07:23, azurIt wrote:
> Any news? Thnx!

Sorry, but I didn't get to this one yet.

>
> azur
>
>
>
> ______________________________________________________________
> > Od: "Michal Hocko" <mhocko@suse.cz>
> > Komu: azurIt <azurit@pobox.sk>
> > Dátum: 30.12.2012 12:08
> > Predmet: Re: [PATCH for 3.2.34] memcg: do not trigger OOM from add_to_page_cache_locked
> >
> > CC: linux-kernel@vger.kernel.org, linux-mm@kvack.org, "cgroups mailinglist" <cgroups@vger.kernel.org>, "KAMEZAWA Hiroyuki" <kamezawa.hiroyu@jp.fujitsu.com>, "Johannes Weiner" <hannes@cmpxchg.org>
> >On Sun 30-12-12 02:09:47, azurIt wrote:
> >> >which suggests that the patch is incomplete and that I am blind :/
> >> >mem_cgroup_cache_charge calls __mem_cgroup_try_charge for the page cache
> >> >and that one doesn't check GFP_MEMCG_NO_OOM. So you need the following
> >> >follow-up patch on top of the one you already have (which should catch
> >> >all the remaining cases).
> >> >Sorry about that...
> >>
> >>
> >> This was, again, killing my MySQL server (search for "(mysqld)"):
> >> http://www.watchdog.sk/lkml/oom_mysqld5
> >
> >grep "Kill process" oom_mysqld5
> >Dec 30 01:53:34 server01 kernel: [ 367.061801] Memory cgroup out of memory: Kill process 5512 (apache2) score 716 or sacrifice child
> >Dec 30 01:53:35 server01 kernel: [ 367.338024] Memory cgroup out of memory: Kill process 5517 (apache2) score 718 or sacrifice child
> >Dec 30 01:53:35 server01 kernel: [ 367.747888] Memory cgroup out of memory: Kill process 5513 (apache2) score 721 or sacrifice child
> >Dec 30 01:53:36 server01 kernel: [ 368.159860] Memory cgroup out of memory: Kill process 5516 (apache2) score 726 or sacrifice child
> >Dec 30 01:53:36 server01 kernel: [ 368.665606] Memory cgroup out of memory: Kill process 5520 (apache2) score 733 or sacrifice child
> >Dec 30 01:53:36 server01 kernel: [ 368.765652] Out of memory: Kill process 1778 (mysqld) score 39 or sacrifice child
> >Dec 30 01:53:36 server01 kernel: [ 369.101753] Memory cgroup out of memory: Kill process 5519 (apache2) score 754 or sacrifice child
> >Dec 30 01:53:37 server01 kernel: [ 369.464262] Memory cgroup out of memory: Kill process 5583 (apache2) score 762 or sacrifice child
> >Dec 30 01:53:37 server01 kernel: [ 369.465017] Out of memory: Kill process 5506 (apache2) score 18 or sacrifice child
> >Dec 30 01:53:37 server01 kernel: [ 369.574932] Memory cgroup out of memory: Kill process 5523 (apache2) score 759 or sacrifice child
> >
> >So your mysqld has been killed by the global OOM not memcg. But why when
> >you seem to be perfectly fine regarding memory? I guess the following
> >backtrace is relevant:
> >Dec 30 01:53:36 server01 kernel: [ 368.569720] DMA: 0*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15912kB
> >Dec 30 01:53:36 server01 kernel: [ 368.570447] DMA32: 9*4kB 10*8kB 8*16kB 6*32kB 5*64kB 6*128kB 4*256kB 2*512kB 3*1024kB 3*2048kB 613*4096kB = 2523636kB
> >Dec 30 01:53:36 server01 kernel: [ 368.571175] Normal: 5*4kB 2060*8kB 4122*16kB 2550*32kB 2667*64kB 722*128kB 197*256kB 68*512kB 15*1024kB 4*2048kB 1855*4096kB = 8134036kB
> >Dec 30 01:53:36 server01 kernel: [ 368.571906] 308964 total pagecache pages
> >Dec 30 01:53:36 server01 kernel: [ 368.572023] 0 pages in swap cache
> >Dec 30 01:53:36 server01 kernel: [ 368.572140] Swap cache stats: add 0, delete 0, find 0/0
> >Dec 30 01:53:36 server01 kernel: [ 368.572260] Free swap = 0kB
> >Dec 30 01:53:36 server01 kernel: [ 368.572375] Total swap = 0kB
> >Dec 30 01:53:36 server01 kernel: [ 368.597836] apache2 invoked oom-killer: gfp_mask=0x0, order=0, oom_adj=0, oom_score_adj=0
> >Dec 30 01:53:36 server01 kernel: [ 368.598034] apache2 cpuset=uid mems_allowed=0
> >Dec 30 01:53:36 server01 kernel: [ 368.598152] Pid: 5385, comm: apache2 Not tainted 3.2.35-grsec #1
> >Dec 30 01:53:36 server01 kernel: [ 368.598273] Call Trace:
> >Dec 30 01:53:36 server01 kernel: [ 368.598396] [<ffffffff810cc89e>] dump_header+0x7e/0x1e0
> >Dec 30 01:53:36 server01 kernel: [ 368.598516] [<ffffffff810cc79f>] ? find_lock_task_mm+0x2f/0x70
> >Dec 30 01:53:36 server01 kernel: [ 368.598638] [<ffffffff810ccd65>] oom_kill_process+0x85/0x2a0
> >Dec 30 01:53:36 server01 kernel: [ 368.598759] [<ffffffff810cd415>] out_of_memory+0xe5/0x200
> >Dec 30 01:53:36 server01 kernel: [ 368.598880] [<ffffffff810cd5ed>] pagefault_out_of_memory+0xbd/0x110
> >Dec 30 01:53:36 server01 kernel: [ 368.599006] [<ffffffff81026e96>] mm_fault_error+0xb6/0x1a0
> >Dec 30 01:53:36 server01 kernel: [ 368.599127] [<ffffffff8102736e>] do_page_fault+0x3ee/0x460
> >Dec 30 01:53:36 server01 kernel: [ 368.599250] [<ffffffff81131ccf>] ? mntput+0x1f/0x30
> >Dec 30 01:53:36 server01 kernel: [ 368.599371] [<ffffffff811134e6>] ? fput+0x156/0x200
> >Dec 30 01:53:36 server01 kernel: [ 368.599496] [<ffffffff815b567f>] page_fault+0x1f/0x30
> >
> >This would suggest that an unexpected ENOMEM leaked during page fault
> >path. I do not see which one could that be because you said THP
> >(CONFIG_TRANSPARENT_HUGEPAGE) are disabled (and the other patch I have
> >mentioned in the thread should fix that issue - btw. the patch is
> >already scheduled for stable tree).
> > __do_fault, do_anonymous_page and do_wp_page call
> >mem_cgroup_newpage_charge with GFP_KERNEL which means that
> >we do memcg OOM and never return ENOMEM. do_swap_page calls
> >mem_cgroup_try_charge_swapin with GFP_KERNEL as well.
> >
> >I might have missed something but I will not get to look closer before
> >2nd January.
> >--
> >Michal Hocko
> >SUSE Labs
> >
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2013-01-25 18:21    [W:0.130 / U:2.808 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site