Messages in this thread | | | Subject | Trying to understand OOM killer | From | Paul Menzel <> | Date | Sun, 12 Feb 2017 14:47:13 +0100 |
| |
Dear Linux folks,
since some time, at Linux 4.8, 4.9, and 4.10-rc6, the OOM kicks in on a 8 GB machine.
``` Feb 12 08:21:50 asrocke350m1 kernel: updatedb.mlocat invoked oom-killer: gfp_mask=0x16040d0(GFP_TEMPORARY|__GFP_COMP|__GFP_NOTRACK), nodemask= Feb 12 08:21:50 asrocke350m1 kernel: updatedb.mlocat cpuset=/ mems_allowed=0 Feb 12 08:21:50 asrocke350m1 kernel: CPU: 1 PID: 2314 Comm: updatedb.mlocat Tainted: G C 4.10.0-rc6-686-pae #1 Debian 4.10~rc6-1~ Feb 12 08:21:50 asrocke350m1 kernel: Hardware name: ASROCK E350M1/E350M1, BIOS 4.5-964-gd96669e9db 02/11/2017 Feb 12 08:21:51 asrocke350m1 kernel: Call Trace: Feb 12 08:21:51 asrocke350m1 kernel: ? dump_stack+0x55/0x73 Feb 12 08:21:51 asrocke350m1 kernel: ? dump_header+0x64/0x1ab Feb 12 08:21:52 asrocke350m1 kernel: ? ___ratelimit+0x9f/0x100 Feb 12 08:21:52 asrocke350m1 kernel: ? oom_kill_process+0x221/0x3e0 Feb 12 08:21:52 asrocke350m1 kernel: ? has_capability_noaudit+0x1a/0x30 Feb 12 08:21:52 asrocke350m1 kernel: ? oom_badness.part.13+0xd7/0x150 Feb 12 08:21:52 asrocke350m1 kernel: ? out_of_memory+0xe4/0x290 Feb 12 08:21:52 asrocke350m1 kernel: ? __alloc_pages_nodemask+0xab8/0xbc0 Feb 12 08:21:52 asrocke350m1 kernel: ? xfs_init_local_fork+0x8a/0xd0 [xfs] Feb 12 08:21:52 asrocke350m1 kernel: ? cache_grow_begin.isra.60+0x75/0x510 Feb 12 08:21:52 asrocke350m1 kernel: ? xfs_buf_rele+0x43/0x2e0 [xfs] Feb 12 08:21:52 asrocke350m1 kernel: ? kmem_cache_alloc+0x1fa/0x530 Feb 12 08:21:52 asrocke350m1 kernel: ? __d_alloc+0x23/0x180 Feb 12 08:21:52 asrocke350m1 kernel: ? d_alloc+0x18/0x80 Feb 12 08:21:52 asrocke350m1 kernel: ? d_alloc_parallel+0x47/0x450 Feb 12 08:21:52 asrocke350m1 kernel: ? d_splice_alias+0x10d/0x3a0 Feb 12 08:21:53 asrocke350m1 kernel: ? lockref_get_not_dead+0x8/0x40 Feb 12 08:21:53 asrocke350m1 kernel: ? unlazy_walk+0xf9/0x1a0 Feb 12 08:21:53 asrocke350m1 kernel: ? lookup_slow+0x5e/0x140 Feb 12 08:21:53 asrocke350m1 kernel: ? walk_component+0x1b4/0x350 Feb 12 08:21:53 asrocke350m1 kernel: ? path_lookupat+0x49/0xe0 Feb 12 08:21:53 asrocke350m1 kernel: ? filename_lookup+0x99/0x190 Feb 12 08:21:53 asrocke350m1 kernel: ? __check_object_size+0x9e/0x11c Feb 12 08:21:53 asrocke350m1 kernel: ? strncpy_from_user+0x39/0x140 Feb 12 08:21:53 asrocke350m1 kernel: ? getname_flags+0x55/0x1a0 Feb 12 08:21:53 asrocke350m1 kernel: ? vfs_fstatat+0x60/0xb0 Feb 12 08:21:53 asrocke350m1 kernel: ? SyS_lstat64+0x2d/0x50 Feb 12 08:21:53 asrocke350m1 kernel: ? sys_sync+0x9d/0xa0 Feb 12 08:21:53 asrocke350m1 kernel: ? SyS_poll+0x6b/0x110 Feb 12 08:21:53 asrocke350m1 kernel: ? do_fast_syscall_32+0x8a/0x150 Feb 12 08:21:53 asrocke350m1 kernel: ? entry_SYSENTER_32+0x4e/0x7c Feb 12 08:21:53 asrocke350m1 kernel: Mem-Info: Feb 12 08:21:53 asrocke350m1 kernel: active_anon:119893 inactive_anon:17678 isolated_anon:0 active_file:31461 inactive_file:219091 isolated_file:0 unevictable:21 dirty:0 writeback:0 unstable:0 slab_reclaimable:127609 slab_unreclaimable:9519 mapped:63113 shmem:6177 pagetables:1601 bounce:0 free:1381579 free_pcp:583 free_cma:0 Feb 12 08:21:53 asrocke350m1 kernel: Node 0 active_anon:479572kB inactive_anon:70712kB active_file:125844kB inactive_file:876364kB unevictable Feb 12 08:21:53 asrocke350m1 kernel: DMA free:3840kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:0kB inactiv Feb 12 08:21:53 asrocke350m1 kernel: lowmem_reserve[]: 0 763 7663 7663 Feb 12 08:21:53 asrocke350m1 kernel: Normal free:38764kB min:38828kB low:48532kB high:58236kB active_anon:0kB inactive_anon:0kB active_file:16 Feb 12 08:21:53 asrocke350m1 kernel: lowmem_reserve[]: 0 0 55201 55201 Feb 12 08:21:53 asrocke350m1 kernel: HighMem free:5483712kB min:512kB low:88240kB high:175968kB active_anon:479572kB inactive_anon:70712kB act Feb 12 08:21:54 asrocke350m1 kernel: lowmem_reserve[]: 0 0 0 0 Feb 12 08:21:54 asrocke350m1 kernel: DMA: 0*4kB 42*8kB (UE) 69*16kB (UE) 7*32kB (UE) 10*64kB (UE) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB Feb 12 08:21:54 asrocke350m1 kernel: Normal: 17*4kB (UME) 583*8kB (UME) 1983*16kB (UE) 72*32kB (ME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0* Feb 12 08:21:54 asrocke350m1 kernel: HighMem: 2156*4kB (UM) 1334*8kB (UM) 2760*16kB (UM) 2087*32kB (UM) 1274*64kB (UM) 491*128kB (UM) 234*256k Feb 12 08:21:54 asrocke350m1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 12 08:21:54 asrocke350m1 kernel: 256720 total pagecache pages Feb 12 08:21:54 asrocke350m1 kernel: 0 pages in swap cache Feb 12 08:21:54 asrocke350m1 kernel: Swap cache stats: add 0, delete 0, find 0/0 Feb 12 08:21:54 asrocke350m1 kernel: Free swap = 4194300kB Feb 12 08:21:54 asrocke350m1 kernel: Total swap = 4194300kB Feb 12 08:21:54 asrocke350m1 kernel: 1994197 pages RAM Feb 12 08:21:54 asrocke350m1 kernel: 1766457 pages HighMem/MovableOnly Feb 12 08:21:54 asrocke350m1 kernel: 22689 pages reserved Feb 12 08:21:54 asrocke350m1 kernel: 0 pages hwpoisoned Feb 12 08:21:54 asrocke350m1 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Feb 12 08:21:54 asrocke350m1 kernel: [ 388] 0 388 18525 8958 32 3 0 0 systemd-journal Feb 12 08:21:54 asrocke350m1 kernel: [ 416] 0 416 5467 391 7 3 0 0 lvmetad Feb 12 08:21:54 asrocke350m1 kernel: [ 429] 0 429 4034 1047 8 3 0 -1000 systemd-udevd Feb 12 08:21:54 asrocke350m1 kernel: [ 597] 0 597 835 573 5 3 0 0 mdadm Feb 12 08:21:54 asrocke350m1 kernel: [ 825] 134 825 4244 992 9 3 0 0 systemd-timesyn Feb 12 08:21:54 asrocke350m1 kernel: [ 835] 0 835 1428 935 6 3 0 0 smartd Feb 12 08:21:54 asrocke350m1 kernel: [ 836] 0 836 1313 719 6 3 0 0 cron Feb 12 08:21:54 asrocke350m1 kernel: [ 838] 0 838 1038 564 6 3 0 0 anacron Feb 12 08:21:54 asrocke350m1 kernel: [ 840] 0 840 5883 751 10 3 0 0 rsyslogd Feb 12 08:21:54 asrocke350m1 kernel: [ 846] 0 846 1108 410 7 3 0 0 irexec Feb 12 08:21:54 asrocke350m1 kernel: [ 851] 0 851 9931 1654 13 3 0 0 accounts-daemon Feb 12 08:21:54 asrocke350m1 kernel: [ 853] 104 853 1699 1110 8 3 0 -900 dbus-daemon Feb 12 08:21:54 asrocke350m1 kernel: [ 867] 105 867 1563 814 6 3 0 0 avahi-daemon Feb 12 08:21:54 asrocke350m1 kernel: [ 868] 0 868 23658 4150 26 3 0 0 NetworkManager Feb 12 08:21:54 asrocke350m1 kernel: [ 869] 0 869 1128 420 6 3 0 0 lircmd Feb 12 08:21:54 asrocke350m1 kernel: [ 870] 0 870 889 568 5 3 0 0 atd Feb 12 08:21:54 asrocke350m1 kernel: [ 878] 0 878 1853 1147 7 3 0 0 systemd-logind Feb 12 08:21:54 asrocke350m1 kernel: [ 879] 0 879 10954 2159 16 3 0 0 ModemManager Feb 12 08:21:54 asrocke350m1 kernel: [ 887] 0 887 558 18 5 3 0 0 minissdpd Feb 12 08:21:54 asrocke350m1 kernel: [ 893] 105 893 1563 73 6 3 0 0 avahi-daemon Feb 12 08:21:54 asrocke350m1 kernel: [ 895] 129 895 11545 3234 17 3 0 0 colord Feb 12 08:21:54 asrocke350m1 kernel: [ 904] 0 904 9909 1981 15 3 0 0 polkitd Feb 12 08:21:54 asrocke350m1 kernel: [ 923] 0 923 94106 6465 51 3 0 0 libvirtd Feb 12 08:21:54 asrocke350m1 kernel: [ 926] 0 926 2616 1283 8 3 0 -1000 sshd Feb 12 08:21:54 asrocke350m1 kernel: [ 961] 0 961 10354 1870 14 3 0 0 gdm3 Feb 12 08:21:54 asrocke350m1 kernel: [ 973] 0 973 7965 1994 13 3 0 0 gdm-session-wor Feb 12 08:21:54 asrocke350m1 kernel: [ 1192] 0 1192 1207 758 7 3 0 0 lircd Feb 12 08:21:54 asrocke350m1 kernel: [ 1195] 0 1195 1129 431 6 3 0 0 lircd-uinput Feb 12 08:21:54 asrocke350m1 kernel: [ 1252] 122 1252 8304 7454 21 3 0 0 tor Feb 12 08:21:55 asrocke350m1 kernel: [ 1255] 101 1255 2813 702 8 3 0 0 exim4 Feb 12 08:21:55 asrocke350m1 kernel: [ 1278] 0 1278 2856 1758 9 3 0 0 wpa_supplicant Feb 12 08:21:55 asrocke350m1 kernel: [ 1280] 0 1280 2117 1352 8 3 0 0 apache2 Feb 12 08:21:55 asrocke350m1 kernel: [ 1288] 33 1288 811 39 5 3 0 0 htcacheclean Feb 12 08:21:55 asrocke350m1 kernel: [ 1289] 130 1289 2369 1407 8 3 0 0 systemd Feb 12 08:21:55 asrocke350m1 kernel: [ 1290] 130 1290 2745 392 9 3 0 0 (sd-pam) Feb 12 08:21:55 asrocke350m1 kernel: [ 1292] 130 1292 6945 1251 10 3 0 0 gdm-wayland-ses Feb 12 08:21:55 asrocke350m1 kernel: [ 1294] 130 1294 1590 1002 7 3 0 0 dbus-daemon Feb 12 08:21:55 asrocke350m1 kernel: [ 1311] 130 1311 19475 3265 23 3 0 0 gnome-session-b Feb 12 08:21:55 asrocke350m1 kernel: [ 1374] 130 1374 208428 26571 111 3 0 0 gnome-shell Feb 12 08:21:55 asrocke350m1 kernel: [ 1379] 0 1379 21121 2547 18 3 0 0 upowerd Feb 12 08:21:55 asrocke350m1 kernel: [ 1405] 130 1405 28029 9140 46 3 0 0 Xwayland Feb 12 08:21:55 asrocke350m1 kernel: [ 1411] 130 1411 11342 1415 12 3 0 0 at-spi-bus-laun Feb 12 08:21:55 asrocke350m1 kernel: [ 1416] 130 1416 1564 873 7 3 0 0 dbus-daemon Feb 12 08:21:55 asrocke350m1 kernel: [ 1418] 130 1418 7592 1544 11 3 0 0 at-spi2-registr Feb 12 08:21:55 asrocke350m1 kernel: [ 1425] 130 1425 222202 2441 21 3 0 0 pulseaudio Feb 12 08:21:55 asrocke350m1 kernel: [ 1440] 0 1440 11899 2862 18 3 0 0 packagekitd Feb 12 08:21:55 asrocke350m1 kernel: [ 1441] 130 1441 115081 12067 68 3 0 0 gnome-settings- Feb 12 08:21:55 asrocke350m1 kernel: [ 1480] 0 1480 7986 2033 14 3 0 0 gdm-session-wor Feb 12 08:21:55 asrocke350m1 kernel: [ 1486] 1000 1486 2369 1371 8 3 0 0 systemd Feb 12 08:21:55 asrocke350m1 kernel: [ 1487] 1000 1487 7355 402 12 3 0 0 (sd-pam) Feb 12 08:21:55 asrocke350m1 kernel: [ 1491] 1000 1491 10023 1922 13 3 0 0 gnome-keyring-d Feb 12 08:21:55 asrocke350m1 kernel: [ 1494] 1000 1494 7424 1403 11 3 0 0 gdm-x-session Feb 12 08:21:55 asrocke350m1 kernel: [ 1496] 1000 1496 33920 14489 59 3 0 0 Xorg Feb 12 08:21:55 asrocke350m1 kernel: [ 1500] 1000 1500 1589 997 7 3 0 0 dbus-daemon Feb 12 08:21:55 asrocke350m1 kernel: [ 1505] 1000 1505 14890 7121 24 3 0 0 awesome Feb 12 08:21:55 asrocke350m1 kernel: [ 1534] 1000 1534 4310 1673 10 3 0 0 arbtt-capture Feb 12 08:21:55 asrocke350m1 kernel: [ 1628] 1000 1628 1180 54 6 3 0 0 ssh-agent Feb 12 08:21:55 asrocke350m1 kernel: [ 1635] 1000 1635 11361 1459 12 3 0 0 at-spi-bus-laun Feb 12 08:21:55 asrocke350m1 kernel: [ 1640] 1000 1640 1564 873 6 3 0 0 dbus-daemon Feb 12 08:21:55 asrocke350m1 kernel: [ 1644] 1000 1644 7594 1294 13 3 0 0 at-spi2-registr Feb 12 08:21:55 asrocke350m1 kernel: [ 1646] 1000 1646 21040 8672 36 3 0 0 gnome-terminal- Feb 12 08:21:55 asrocke350m1 kernel: [ 1653] 1000 1653 10027 1331 15 3 0 0 gvfsd Feb 12 08:21:55 asrocke350m1 kernel: [ 1658] 1000 1658 13148 1359 15 3 0 0 gvfsd-fuse Feb 12 08:21:55 asrocke350m1 kernel: [ 1672] 1000 1672 1773 1222 7 3 0 0 bash Feb 12 08:21:55 asrocke350m1 kernel: [ 1681] 1000 1681 15803 13913 36 3 0 0 gdb Feb 12 08:21:55 asrocke350m1 kernel: [ 1689] 1000 1689 1759 1205 6 3 0 0 bash Feb 12 08:21:55 asrocke350m1 kernel: [ 1700] 1000 1700 1759 1211 7 3 0 0 bash Feb 12 08:21:55 asrocke350m1 kernel: [ 1713] 1000 1713 19912 7108 29 3 0 0 nm-applet Feb 12 08:21:55 asrocke350m1 kernel: [ 1724] 1000 1724 10075 1350 12 3 0 0 gnome-keyring-d Feb 12 08:21:55 asrocke350m1 kernel: [ 1730] 0 1730 2026 927 7 3 0 0 dhclient Feb 12 08:21:55 asrocke350m1 kernel: [ 1925] 0 1925 555 321 5 3 0 0 run-parts Feb 12 08:21:56 asrocke350m1 kernel: [ 2208] 33 2208 2123 1124 8 3 0 0 apache2 Feb 12 08:21:56 asrocke350m1 kernel: [ 2209] 33 2209 2123 795 7 3 0 0 apache2 Feb 12 08:21:56 asrocke350m1 kernel: [ 2210] 33 2210 2123 795 7 3 0 0 apache2 Feb 12 08:21:57 asrocke350m1 kernel: [ 2211] 33 2211 2123 795 7 3 0 0 apache2 Feb 12 08:21:58 asrocke350m1 kernel: [ 2212] 33 2212 2123 795 7 3 0 0 apache2 Feb 12 08:21:59 asrocke350m1 kernel: [ 2213] 33 2213 2123 795 7 3 0 0 apache2 Feb 12 08:21:59 asrocke350m1 kernel: [ 2227] 0 2227 3831 1969 12 3 0 0 cupsd Feb 12 08:21:59 asrocke350m1 kernel: [ 2308] 0 2308 1307 712 6 3 0 0 mlocate Feb 12 08:22:00 asrocke350m1 kernel: [ 2313] 0 2313 1008 160 6 3 0 0 flock Feb 12 08:22:01 asrocke350m1 kernel: [ 2314] 0 2314 1183 726 7 3 0 0 updatedb.mlocat Feb 12 08:22:01 asrocke350m1 kernel: [ 2780] 1000 2780 2559 1409 8 3 0 0 ssh Feb 12 08:22:01 asrocke350m1 kernel: [ 2789] 1000 2789 1773 1264 7 3 0 0 bash Feb 12 08:22:01 asrocke350m1 kernel: [ 2798] 1000 2798 208137 83634 291 3 0 0 firefox-esr Feb 12 08:22:01 asrocke350m1 kernel: [ 2807] 1000 2807 1760 1253 7 3 0 0 bash Feb 12 08:22:01 asrocke350m1 kernel: [ 2833] 1000 2833 3332 1638 11 3 0 0 gconfd-2 Feb 12 08:22:01 asrocke350m1 kernel: [ 2883] 1000 2883 2944 2168 8 3 0 0 vim Feb 12 08:22:02 asrocke350m1 kernel: [ 2895] 1000 2895 1784 1327 7 3 0 0 bash Feb 12 08:22:03 asrocke350m1 kernel: [ 4298] 1000 4298 3444 2800 10 3 0 0 debcheckout Feb 12 08:22:03 asrocke350m1 kernel: [ 4302] 1000 4302 1898 998 7 3 0 0 git Feb 12 08:22:03 asrocke350m1 kernel: [ 4303] 1000 4303 11054 6508 22 3 0 0 git-remote-http Feb 12 08:22:04 asrocke350m1 kernel: [ 4306] 1000 4306 4232 983 8 3 0 0 git Feb 12 08:22:04 asrocke350m1 kernel: [ 4309] 1000 4309 2621 1522 8 3 0 0 git Feb 12 08:22:04 asrocke350m1 kernel: [ 4322] 0 4322 1408 829 6 3 0 0 inetd Feb 12 08:22:05 asrocke350m1 kernel: Out of memory: Kill process 2798 (firefox-esr) score 27 or sacrifice child Feb 12 08:22:05 asrocke350m1 kernel: Killed process 2798 (firefox-esr) total-vm:832548kB, anon-rss:248168kB, file-rss:86300kB, shmem-rss:68kB Feb 12 08:22:06 asrocke350m1 kernel: perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 ```
The sum of the RSS values is 312,260. According to the article [1], one page is 4 KB in size. That make it less then 1.3 GB.
``` $ more /proc/meminfo # after OOM run MemTotal: 7886032 kB MemFree: 1613032 kB MemAvailable: 4510132 kB Buffers: 160536 kB Cached: 3103908 kB SwapCached: 0 kB Active: 4004748 kB Inactive: 1629624 kB Active(anon): 2255340 kB Inactive(anon): 158560 kB Active(file): 1749408 kB Inactive(file): 1471064 kB Unevictable: 232 kB Mlocked: 232 kB HighTotal: 7065828 kB HighFree: 1555764 kB LowTotal: 820204 kB LowFree: 57268 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 12 kB Writeback: 0 kB AnonPages: 2370220 kB Mapped: 419968 kB Shmem: 43972 kB Slab: 460304 kB SReclaimable: 423320 kB SUnreclaim: 36984 kB KernelStack: 4224 kB PageTables: 14580 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 8137316 kB Committed_AS: 5379296 kB VmallocTotal: 122880 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 10232 kB DirectMap2M: 901120 kB ```
So I wonder, why the OOM killer kicked in at all.
Hints and insight is appreciated.
Thanks,
Paul
[1] http://careers.directi.com/display/tu/Understanding+and+optimizing+Memory+utilization[unhandled content-type:application/pgp-signature] | |