Messages in this thread | | | Subject | Re: OOM-Killer kills too much with 2.6.32.2 | Date | Mon, 25 Jan 2010 23:12:38 +0100 | From | "Roman Jarosz" <> |
| |
On Sat, 23 Jan 2010 01:40:17 +0100, David Rientjes <rientjes@google.com> wrote:
> On Thu, 14 Jan 2010, Roman Jarosz wrote: > >> Hi, >> >> since kernel 2.6.32.2 (also tried 2.6.32.3) I get a lot of oom-killer >> kills >> when I do hard disk intensive tasks (mainly in VirtualBox which is >> running >> Windows XP) and IMHO it kills processes even if I have a lot of free >> memory. >> >> Is this a known bug? I have self compiled kernel so I can try patches. >> >> Regards >> Roman Jarosz >> >> PS. Please CC me. >> >> Jan 7 12:39:27 kedge kernel: X invoked oom-killer: gfp_mask=0x0, >> order=0, >> oom_adj=0 > > The gfp_mask of 0x0 and order of 0 indicate these are triggered by > pagefaults that end up returning VM_FAULT_OOM. Prior to 2.6.29, the > current task (X in all cases from your log) would have been SIGKILLed; we > now call the oom killer instead of kill a memory hogging task so that the > fault is retried with more success. If you've upgraded from a 2.6.29 or > later kernel and are only now experiencing these errors, it may indicate > a > regression in VM that we need to investigate (and, if so, you may want to > try merging f50de2d38 from 2.6.33 to see if it helps keep more > ZONE_NORMAL > memory available so that such drastic measures aren't necessary).
I only have logs since 2.6.31-rc7 and there is one oom-kill in 2.6.32-rc7 and then the kills in 2.6.32.2.
Nov 25 13:49:04 kedge kernel: X invoked oom-killer: gfp_mask=0x0, order=0, oom_adj=0 Nov 25 13:49:04 kedge kernel: Pid: 1883, comm: X Not tainted 2.6.32-rc7 #1 Nov 25 13:49:04 kedge kernel: Call Trace: Nov 25 13:49:04 kedge kernel: [<ffffffff8107a66d>] ? 0xffffffff8107a66d Nov 25 13:49:04 kedge kernel: [<ffffffff8107a953>] ? 0xffffffff8107a953 Nov 25 13:49:04 kedge kernel: [<ffffffff8107aac7>] ? 0xffffffff8107aac7 Nov 25 13:49:04 kedge kernel: [<ffffffff8155453f>] ? 0xffffffff8155453f Nov 25 13:49:04 kedge kernel: Mem-Info: Nov 25 13:49:04 kedge kernel: DMA per-cpu: Nov 25 13:49:04 kedge kernel: CPU 0: hi: 0, btch: 1 usd: 0 Nov 25 13:49:04 kedge kernel: CPU 1: hi: 0, btch: 1 usd: 0 Nov 25 13:49:04 kedge kernel: DMA32 per-cpu: Nov 25 13:49:04 kedge kernel: CPU 0: hi: 186, btch: 31 usd: 178 Nov 25 13:49:04 kedge kernel: CPU 1: hi: 186, btch: 31 usd: 174 Nov 25 13:49:04 kedge kernel: Normal per-cpu: Nov 25 13:49:04 kedge kernel: CPU 0: hi: 186, btch: 31 usd: 153 Nov 25 13:49:04 kedge kernel: CPU 1: hi: 186, btch: 31 usd: 81 Nov 25 13:49:04 kedge kernel: active_anon:135012 inactive_anon:77065 isolated_anon:0 Nov 25 13:49:04 kedge kernel: active_file:332368 inactive_file:411112 isolated_file:32 Nov 25 13:49:04 kedge kernel: unevictable:0 dirty:139692 writeback:2395 unstable:0 Nov 25 13:49:04 kedge kernel: free:6869 slab_reclaimable:15618 slab_unreclaimable:7026 Nov 25 13:49:04 kedge kernel: mapped:31999 shmem:26786 pagetables:4497 bounce:0 Nov 25 13:49:04 kedge kernel: DMA free:15776kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:1 60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15344kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_rec laimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaima ble? no Nov 25 13:49:04 kedge kernel: lowmem_reserve[]: 0 2990 3937 3937 Nov 25 13:49:04 kedge kernel: DMA32 free:9856kB min:6084kB low:7604kB high:9124kB active_anon:327516kB inactive_anon:87488kB active_file:112 2804kB inactive_file:1374360kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3062688kB mlocked:0kB dirty:440312kB writeback: 5556kB mapped:24652kB shmem:57952kB slab_reclaimable:42492kB slab_unreclaimable:10528kB kernel_stack:472kB pagetables:3464kB unstable:0kB bo unce:0kB writeback_tmp:0kB pages_scanned:384 all_unreclaimable? no Nov 25 13:49:04 kedge kernel: lowmem_reserve[]: 0 0 946 946 Nov 25 13:49:04 kedge kernel: Normal free:1844kB min:1924kB low:2404kB high:2884kB active_anon:212532kB inactive_anon:220772kB active_file:2 06668kB inactive_file:269928kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:969600kB mlocked:0kB dirty:118456kB writeback :4024kB mapped:103344kB shmem:49192kB slab_reclaimable:19980kB slab_unreclaimable:17568kB kernel_stack:1608kB pagetables:14524kB unstable:0k B bounce:0kB writeback_tmp:0kB pages_scanned:448 all_unreclaimable? no Nov 25 13:49:04 kedge kernel: lowmem_reserve[]: 0 0 0 0 Nov 25 13:49:04 kedge kernel: DMA: 0*4kB 2*8kB 1*16kB 2*32kB 1*64kB 2*128kB 2*256kB 1*512kB 2*1024kB 2*2048kB 2*4096kB = 15776kB Nov 25 13:49:04 kedge kernel: DMA32: 300*4kB 4*8kB 29*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 9856kB Nov 25 13:49:04 kedge kernel: Normal: 461*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1844kB Nov 25 13:49:04 kedge kernel: 754208 total pagecache pages Nov 25 13:49:04 kedge kernel: 64 pages in swap cache Nov 25 13:49:04 kedge kernel: Swap cache stats: add 357, delete 293, find 2/8 Nov 25 13:49:04 kedge kernel: Free swap = 2006888kB Nov 25 13:49:04 kedge kernel: Total swap = 2008116kB Nov 25 13:49:04 kedge kernel: 1032192 pages RAM Nov 25 13:49:04 kedge kernel: 36869 pages reserved Nov 25 13:49:04 kedge kernel: 377334 pages shared Nov 25 13:49:04 kedge kernel: 705565 pages non-shared Nov 25 13:49:04 kedge kernel: Out of memory: kill process 6816 (services.exe) score 1652318 or a child Nov 25 13:49:04 kedge kernel: Killed process 6818 (winedevice.exe)
>> Jan 7 12:39:27 kedge kernel: Pid: 1954, comm: X Not tainted 2.6.32.2 #1 >> Jan 7 12:39:27 kedge kernel: Call Trace: >> Jan 7 12:39:27 kedge kernel: [<ffffffff8107b17d>] ? 0xffffffff8107b17d >> Jan 7 12:39:27 kedge kernel: [<ffffffff8107b463>] ? 0xffffffff8107b463 >> Jan 7 12:39:27 kedge kernel: [<ffffffff8107b5d7>] ? 0xffffffff8107b5d7 >> Jan 7 12:39:27 kedge kernel: [<ffffffff815581df>] ? 0xffffffff815581df > > Can you find out what these symbols are?
Can I somehow get the symbols without recompiling kernel? I have the source tree with *.o files and other stuff which was created during compilation.
I've also got the kill in 2.6.33-rc5, what do I have to do to get the symbols? Is "Load all symbols for debugging/ksymoops (KALLSYMS)" and "Compile the kernel with frame pointers (FRAME_POINTER)" enough?
Jan 25 15:51:45 kedge kernel: X invoked oom-killer: gfp_mask=0x0, order=0, oom_adj=0 Jan 25 15:51:45 kedge kernel: Pid: 1904, comm: X Not tainted 2.6.33-rc5 #1 Jan 25 15:51:45 kedge kernel: Call Trace: Jan 25 15:51:45 kedge kernel: [<ffffffff8107e1a4>] 0xffffffff8107e1a4 Jan 25 15:51:45 kedge kernel: [<ffffffff8107e306>] 0xffffffff8107e306 Jan 25 15:51:45 kedge kernel: [<ffffffff8107e4f0>] 0xffffffff8107e4f0 Jan 25 15:51:45 kedge kernel: [<ffffffff8107e678>] 0xffffffff8107e678 Jan 25 15:51:45 kedge kernel: [<ffffffff81021e60>] 0xffffffff81021e60 Jan 25 15:51:45 kedge kernel: [<ffffffff810221a8>] 0xffffffff810221a8 Jan 25 15:51:45 kedge kernel: [<ffffffff8156f95f>] 0xffffffff8156f95f Jan 25 15:51:45 kedge kernel: Mem-Info: Jan 25 15:51:45 kedge kernel: DMA per-cpu: Jan 25 15:51:45 kedge kernel: CPU 0: hi: 0, btch: 1 usd: 0 Jan 25 15:51:45 kedge kernel: CPU 1: hi: 0, btch: 1 usd: 0 Jan 25 15:51:45 kedge kernel: DMA32 per-cpu: Jan 25 15:51:45 kedge kernel: CPU 0: hi: 186, btch: 31 usd: 152 Jan 25 15:51:45 kedge kernel: CPU 1: hi: 186, btch: 31 usd: 171 Jan 25 15:51:45 kedge kernel: Normal per-cpu: Jan 25 15:51:45 kedge kernel: CPU 0: hi: 186, btch: 31 usd: 44 Jan 25 15:51:45 kedge kernel: CPU 1: hi: 186, btch: 31 usd: 167 Jan 25 15:51:45 kedge kernel: active_anon:303490 inactive_anon:130718 isolated_anon:32 Jan 25 15:51:45 kedge kernel: active_file:308170 inactive_file:183887 isolated_file:0 Jan 25 15:51:45 kedge kernel: unevictable:0 dirty:172639 writeback:8741 unstable:0 Jan 25 15:51:45 kedge kernel: free:6871 slab_reclaimable:26390 slab_unreclaimable:11635 Jan 25 15:51:45 kedge kernel: mapped:38879 shmem:29781 pagetables:5198 bounce:0 Jan 25 15:51:45 kedge kernel: DMA free:15764kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:156kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15332kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jan 25 15:51:45 kedge kernel: lowmem_reserve[]: 0 2990 3937 3937 Jan 25 15:51:45 kedge kernel: DMA32 free:9832kB min:6084kB low:7604kB high:9124kB active_anon:970168kB inactive_anon:242576kB active_file:1015796kB inactive_file:619228kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:3062688kB mlocked:0kB dirty:584004kB writeback:31268kB mapped:35252kB shmem:43448kB slab_reclaimable:85656kB slab_unreclaimable:25596kB kernel_stack:264kB pagetables:4724kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:238 all_unreclaimable? no Jan 25 15:51:45 kedge kernel: lowmem_reserve[]: 0 0 946 946 Jan 25 15:51:45 kedge kernel: Normal free:1888kB min:1924kB low:2404kB high:2884kB active_anon:243792kB inactive_anon:280296kB active_file:216880kB inactive_file:116164kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:969600kB mlocked:0kB dirty:106552kB writeback:3696kB mapped:120264kB shmem:75676kB slab_reclaimable:19900kB slab_unreclaimable:20928kB kernel_stack:1920kB pagetables:16068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no Jan 25 15:51:45 kedge kernel: lowmem_reserve[]: 0 0 0 0 Jan 25 15:51:45 kedge kernel: DMA: 1*4kB 2*8kB 2*16kB 3*32kB 2*64kB 3*128kB 3*256kB 2*512kB 3*1024kB 3*2048kB 1*4096kB = 15764kB Jan 25 15:51:45 kedge kernel: DMA32: 1108*4kB 1*8kB 2*16kB 0*32kB 0*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 9848kB Jan 25 15:51:45 kedge kernel: Normal: 472*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1888kB Jan 25 15:51:45 kedge kernel: 522113 total pagecache pages Jan 25 15:51:45 kedge kernel: 274 pages in swap cache Jan 25 15:51:45 kedge kernel: Swap cache stats: add 5340, delete 5066, find 0/0 Jan 25 15:51:45 kedge kernel: Free swap = 1986756kB Jan 25 15:51:45 kedge kernel: Total swap = 2008116kB Jan 25 15:51:45 kedge kernel: 1032192 pages RAM Jan 25 15:51:45 kedge kernel: 49813 pages reserved Jan 25 15:51:45 kedge kernel: 821582 pages shared Jan 25 15:51:45 kedge kernel: 260542 pages non-shared Jan 25 15:51:45 kedge kernel: Out of memory: kill process 1746 (hald) score 10058 or a child Jan 25 15:51:45 kedge kernel: Killed process 1747 (hald-runner) vsz:17860kB, anon-rss:24kB, file-rss:1044kB
| |