lkml.org 
[lkml]   [2017]   [Feb]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectTrying to understand OOM killer
From
Date
Dear Linux folks,


since some time, at Linux 4.8, 4.9, and 4.10-rc6, the OOM kicks in on a
8 GB machine.

```
Feb 12 08:21:50 asrocke350m1 kernel: updatedb.mlocat invoked oom-killer: gfp_mask=0x16040d0(GFP_TEMPORARY|__GFP_COMP|__GFP_NOTRACK), nodemask=
Feb 12 08:21:50 asrocke350m1 kernel: updatedb.mlocat cpuset=/ mems_allowed=0
Feb 12 08:21:50 asrocke350m1 kernel: CPU: 1 PID: 2314 Comm: updatedb.mlocat Tainted: G C 4.10.0-rc6-686-pae #1 Debian 4.10~rc6-1~
Feb 12 08:21:50 asrocke350m1 kernel: Hardware name: ASROCK E350M1/E350M1, BIOS 4.5-964-gd96669e9db 02/11/2017
Feb 12 08:21:51 asrocke350m1 kernel: Call Trace:
Feb 12 08:21:51 asrocke350m1 kernel: ? dump_stack+0x55/0x73
Feb 12 08:21:51 asrocke350m1 kernel: ? dump_header+0x64/0x1ab
Feb 12 08:21:52 asrocke350m1 kernel: ? ___ratelimit+0x9f/0x100
Feb 12 08:21:52 asrocke350m1 kernel: ? oom_kill_process+0x221/0x3e0
Feb 12 08:21:52 asrocke350m1 kernel: ? has_capability_noaudit+0x1a/0x30
Feb 12 08:21:52 asrocke350m1 kernel: ? oom_badness.part.13+0xd7/0x150
Feb 12 08:21:52 asrocke350m1 kernel: ? out_of_memory+0xe4/0x290
Feb 12 08:21:52 asrocke350m1 kernel: ? __alloc_pages_nodemask+0xab8/0xbc0
Feb 12 08:21:52 asrocke350m1 kernel: ? xfs_init_local_fork+0x8a/0xd0 [xfs]
Feb 12 08:21:52 asrocke350m1 kernel: ? cache_grow_begin.isra.60+0x75/0x510
Feb 12 08:21:52 asrocke350m1 kernel: ? xfs_buf_rele+0x43/0x2e0 [xfs]
Feb 12 08:21:52 asrocke350m1 kernel: ? kmem_cache_alloc+0x1fa/0x530
Feb 12 08:21:52 asrocke350m1 kernel: ? __d_alloc+0x23/0x180
Feb 12 08:21:52 asrocke350m1 kernel: ? d_alloc+0x18/0x80
Feb 12 08:21:52 asrocke350m1 kernel: ? d_alloc_parallel+0x47/0x450
Feb 12 08:21:52 asrocke350m1 kernel: ? d_splice_alias+0x10d/0x3a0
Feb 12 08:21:53 asrocke350m1 kernel: ? lockref_get_not_dead+0x8/0x40
Feb 12 08:21:53 asrocke350m1 kernel: ? unlazy_walk+0xf9/0x1a0
Feb 12 08:21:53 asrocke350m1 kernel: ? lookup_slow+0x5e/0x140
Feb 12 08:21:53 asrocke350m1 kernel: ? walk_component+0x1b4/0x350
Feb 12 08:21:53 asrocke350m1 kernel: ? path_lookupat+0x49/0xe0
Feb 12 08:21:53 asrocke350m1 kernel: ? filename_lookup+0x99/0x190
Feb 12 08:21:53 asrocke350m1 kernel: ? __check_object_size+0x9e/0x11c
Feb 12 08:21:53 asrocke350m1 kernel: ? strncpy_from_user+0x39/0x140
Feb 12 08:21:53 asrocke350m1 kernel: ? getname_flags+0x55/0x1a0
Feb 12 08:21:53 asrocke350m1 kernel: ? vfs_fstatat+0x60/0xb0
Feb 12 08:21:53 asrocke350m1 kernel: ? SyS_lstat64+0x2d/0x50
Feb 12 08:21:53 asrocke350m1 kernel: ? sys_sync+0x9d/0xa0
Feb 12 08:21:53 asrocke350m1 kernel: ? SyS_poll+0x6b/0x110
Feb 12 08:21:53 asrocke350m1 kernel: ? do_fast_syscall_32+0x8a/0x150
Feb 12 08:21:53 asrocke350m1 kernel: ? entry_SYSENTER_32+0x4e/0x7c
Feb 12 08:21:53 asrocke350m1 kernel: Mem-Info:
Feb 12 08:21:53 asrocke350m1 kernel: active_anon:119893 inactive_anon:17678 isolated_anon:0
active_file:31461 inactive_file:219091 isolated_file:0
unevictable:21 dirty:0 writeback:0 unstable:0
slab_reclaimable:127609 slab_unreclaimable:9519
mapped:63113 shmem:6177 pagetables:1601 bounce:0
free:1381579 free_pcp:583 free_cma:0
Feb 12 08:21:53 asrocke350m1 kernel: Node 0 active_anon:479572kB inactive_anon:70712kB active_file:125844kB inactive_file:876364kB unevictable
Feb 12 08:21:53 asrocke350m1 kernel: DMA free:3840kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:0kB inactiv
Feb 12 08:21:53 asrocke350m1 kernel: lowmem_reserve[]: 0 763 7663 7663
Feb 12 08:21:53 asrocke350m1 kernel: Normal free:38764kB min:38828kB low:48532kB high:58236kB active_anon:0kB inactive_anon:0kB active_file:16
Feb 12 08:21:53 asrocke350m1 kernel: lowmem_reserve[]: 0 0 55201 55201
Feb 12 08:21:53 asrocke350m1 kernel: HighMem free:5483712kB min:512kB low:88240kB high:175968kB active_anon:479572kB inactive_anon:70712kB act
Feb 12 08:21:54 asrocke350m1 kernel: lowmem_reserve[]: 0 0 0 0
Feb 12 08:21:54 asrocke350m1 kernel: DMA: 0*4kB 42*8kB (UE) 69*16kB (UE) 7*32kB (UE) 10*64kB (UE) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB
Feb 12 08:21:54 asrocke350m1 kernel: Normal: 17*4kB (UME) 583*8kB (UME) 1983*16kB (UE) 72*32kB (ME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*
Feb 12 08:21:54 asrocke350m1 kernel: HighMem: 2156*4kB (UM) 1334*8kB (UM) 2760*16kB (UM) 2087*32kB (UM) 1274*64kB (UM) 491*128kB (UM) 234*256k
Feb 12 08:21:54 asrocke350m1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 12 08:21:54 asrocke350m1 kernel: 256720 total pagecache pages
Feb 12 08:21:54 asrocke350m1 kernel: 0 pages in swap cache
Feb 12 08:21:54 asrocke350m1 kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb 12 08:21:54 asrocke350m1 kernel: Free swap = 4194300kB
Feb 12 08:21:54 asrocke350m1 kernel: Total swap = 4194300kB
Feb 12 08:21:54 asrocke350m1 kernel: 1994197 pages RAM
Feb 12 08:21:54 asrocke350m1 kernel: 1766457 pages HighMem/MovableOnly
Feb 12 08:21:54 asrocke350m1 kernel: 22689 pages reserved
Feb 12 08:21:54 asrocke350m1 kernel: 0 pages hwpoisoned
Feb 12 08:21:54 asrocke350m1 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 12 08:21:54 asrocke350m1 kernel: [ 388] 0 388 18525 8958 32 3 0 0 systemd-journal
Feb 12 08:21:54 asrocke350m1 kernel: [ 416] 0 416 5467 391 7 3 0 0 lvmetad
Feb 12 08:21:54 asrocke350m1 kernel: [ 429] 0 429 4034 1047 8 3 0 -1000 systemd-udevd
Feb 12 08:21:54 asrocke350m1 kernel: [ 597] 0 597 835 573 5 3 0 0 mdadm
Feb 12 08:21:54 asrocke350m1 kernel: [ 825] 134 825 4244 992 9 3 0 0 systemd-timesyn
Feb 12 08:21:54 asrocke350m1 kernel: [ 835] 0 835 1428 935 6 3 0 0 smartd
Feb 12 08:21:54 asrocke350m1 kernel: [ 836] 0 836 1313 719 6 3 0 0 cron
Feb 12 08:21:54 asrocke350m1 kernel: [ 838] 0 838 1038 564 6 3 0 0 anacron
Feb 12 08:21:54 asrocke350m1 kernel: [ 840] 0 840 5883 751 10 3 0 0 rsyslogd
Feb 12 08:21:54 asrocke350m1 kernel: [ 846] 0 846 1108 410 7 3 0 0 irexec
Feb 12 08:21:54 asrocke350m1 kernel: [ 851] 0 851 9931 1654 13 3 0 0 accounts-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [ 853] 104 853 1699 1110 8 3 0 -900 dbus-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [ 867] 105 867 1563 814 6 3 0 0 avahi-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [ 868] 0 868 23658 4150 26 3 0 0 NetworkManager
Feb 12 08:21:54 asrocke350m1 kernel: [ 869] 0 869 1128 420 6 3 0 0 lircmd
Feb 12 08:21:54 asrocke350m1 kernel: [ 870] 0 870 889 568 5 3 0 0 atd
Feb 12 08:21:54 asrocke350m1 kernel: [ 878] 0 878 1853 1147 7 3 0 0 systemd-logind
Feb 12 08:21:54 asrocke350m1 kernel: [ 879] 0 879 10954 2159 16 3 0 0 ModemManager
Feb 12 08:21:54 asrocke350m1 kernel: [ 887] 0 887 558 18 5 3 0 0 minissdpd
Feb 12 08:21:54 asrocke350m1 kernel: [ 893] 105 893 1563 73 6 3 0 0 avahi-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [ 895] 129 895 11545 3234 17 3 0 0 colord
Feb 12 08:21:54 asrocke350m1 kernel: [ 904] 0 904 9909 1981 15 3 0 0 polkitd
Feb 12 08:21:54 asrocke350m1 kernel: [ 923] 0 923 94106 6465 51 3 0 0 libvirtd
Feb 12 08:21:54 asrocke350m1 kernel: [ 926] 0 926 2616 1283 8 3 0 -1000 sshd
Feb 12 08:21:54 asrocke350m1 kernel: [ 961] 0 961 10354 1870 14 3 0 0 gdm3
Feb 12 08:21:54 asrocke350m1 kernel: [ 973] 0 973 7965 1994 13 3 0 0 gdm-session-wor
Feb 12 08:21:54 asrocke350m1 kernel: [ 1192] 0 1192 1207 758 7 3 0 0 lircd
Feb 12 08:21:54 asrocke350m1 kernel: [ 1195] 0 1195 1129 431 6 3 0 0 lircd-uinput
Feb 12 08:21:54 asrocke350m1 kernel: [ 1252] 122 1252 8304 7454 21 3 0 0 tor
Feb 12 08:21:55 asrocke350m1 kernel: [ 1255] 101 1255 2813 702 8 3 0 0 exim4
Feb 12 08:21:55 asrocke350m1 kernel: [ 1278] 0 1278 2856 1758 9 3 0 0 wpa_supplicant
Feb 12 08:21:55 asrocke350m1 kernel: [ 1280] 0 1280 2117 1352 8 3 0 0 apache2
Feb 12 08:21:55 asrocke350m1 kernel: [ 1288] 33 1288 811 39 5 3 0 0 htcacheclean
Feb 12 08:21:55 asrocke350m1 kernel: [ 1289] 130 1289 2369 1407 8 3 0 0 systemd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1290] 130 1290 2745 392 9 3 0 0 (sd-pam)
Feb 12 08:21:55 asrocke350m1 kernel: [ 1292] 130 1292 6945 1251 10 3 0 0 gdm-wayland-ses
Feb 12 08:21:55 asrocke350m1 kernel: [ 1294] 130 1294 1590 1002 7 3 0 0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1311] 130 1311 19475 3265 23 3 0 0 gnome-session-b
Feb 12 08:21:55 asrocke350m1 kernel: [ 1374] 130 1374 208428 26571 111 3 0 0 gnome-shell
Feb 12 08:21:55 asrocke350m1 kernel: [ 1379] 0 1379 21121 2547 18 3 0 0 upowerd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1405] 130 1405 28029 9140 46 3 0 0 Xwayland
Feb 12 08:21:55 asrocke350m1 kernel: [ 1411] 130 1411 11342 1415 12 3 0 0 at-spi-bus-laun
Feb 12 08:21:55 asrocke350m1 kernel: [ 1416] 130 1416 1564 873 7 3 0 0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1418] 130 1418 7592 1544 11 3 0 0 at-spi2-registr
Feb 12 08:21:55 asrocke350m1 kernel: [ 1425] 130 1425 222202 2441 21 3 0 0 pulseaudio
Feb 12 08:21:55 asrocke350m1 kernel: [ 1440] 0 1440 11899 2862 18 3 0 0 packagekitd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1441] 130 1441 115081 12067 68 3 0 0 gnome-settings-
Feb 12 08:21:55 asrocke350m1 kernel: [ 1480] 0 1480 7986 2033 14 3 0 0 gdm-session-wor
Feb 12 08:21:55 asrocke350m1 kernel: [ 1486] 1000 1486 2369 1371 8 3 0 0 systemd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1487] 1000 1487 7355 402 12 3 0 0 (sd-pam)
Feb 12 08:21:55 asrocke350m1 kernel: [ 1491] 1000 1491 10023 1922 13 3 0 0 gnome-keyring-d
Feb 12 08:21:55 asrocke350m1 kernel: [ 1494] 1000 1494 7424 1403 11 3 0 0 gdm-x-session
Feb 12 08:21:55 asrocke350m1 kernel: [ 1496] 1000 1496 33920 14489 59 3 0 0 Xorg
Feb 12 08:21:55 asrocke350m1 kernel: [ 1500] 1000 1500 1589 997 7 3 0 0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1505] 1000 1505 14890 7121 24 3 0 0 awesome
Feb 12 08:21:55 asrocke350m1 kernel: [ 1534] 1000 1534 4310 1673 10 3 0 0 arbtt-capture
Feb 12 08:21:55 asrocke350m1 kernel: [ 1628] 1000 1628 1180 54 6 3 0 0 ssh-agent
Feb 12 08:21:55 asrocke350m1 kernel: [ 1635] 1000 1635 11361 1459 12 3 0 0 at-spi-bus-laun
Feb 12 08:21:55 asrocke350m1 kernel: [ 1640] 1000 1640 1564 873 6 3 0 0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1644] 1000 1644 7594 1294 13 3 0 0 at-spi2-registr
Feb 12 08:21:55 asrocke350m1 kernel: [ 1646] 1000 1646 21040 8672 36 3 0 0 gnome-terminal-
Feb 12 08:21:55 asrocke350m1 kernel: [ 1653] 1000 1653 10027 1331 15 3 0 0 gvfsd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1658] 1000 1658 13148 1359 15 3 0 0 gvfsd-fuse
Feb 12 08:21:55 asrocke350m1 kernel: [ 1672] 1000 1672 1773 1222 7 3 0 0 bash
Feb 12 08:21:55 asrocke350m1 kernel: [ 1681] 1000 1681 15803 13913 36 3 0 0 gdb
Feb 12 08:21:55 asrocke350m1 kernel: [ 1689] 1000 1689 1759 1205 6 3 0 0 bash
Feb 12 08:21:55 asrocke350m1 kernel: [ 1700] 1000 1700 1759 1211 7 3 0 0 bash
Feb 12 08:21:55 asrocke350m1 kernel: [ 1713] 1000 1713 19912 7108 29 3 0 0 nm-applet
Feb 12 08:21:55 asrocke350m1 kernel: [ 1724] 1000 1724 10075 1350 12 3 0 0 gnome-keyring-d
Feb 12 08:21:55 asrocke350m1 kernel: [ 1730] 0 1730 2026 927 7 3 0 0 dhclient
Feb 12 08:21:55 asrocke350m1 kernel: [ 1925] 0 1925 555 321 5 3 0 0 run-parts
Feb 12 08:21:56 asrocke350m1 kernel: [ 2208] 33 2208 2123 1124 8 3 0 0 apache2
Feb 12 08:21:56 asrocke350m1 kernel: [ 2209] 33 2209 2123 795 7 3 0 0 apache2
Feb 12 08:21:56 asrocke350m1 kernel: [ 2210] 33 2210 2123 795 7 3 0 0 apache2
Feb 12 08:21:57 asrocke350m1 kernel: [ 2211] 33 2211 2123 795 7 3 0 0 apache2
Feb 12 08:21:58 asrocke350m1 kernel: [ 2212] 33 2212 2123 795 7 3 0 0 apache2
Feb 12 08:21:59 asrocke350m1 kernel: [ 2213] 33 2213 2123 795 7 3 0 0 apache2
Feb 12 08:21:59 asrocke350m1 kernel: [ 2227] 0 2227 3831 1969 12 3 0 0 cupsd
Feb 12 08:21:59 asrocke350m1 kernel: [ 2308] 0 2308 1307 712 6 3 0 0 mlocate
Feb 12 08:22:00 asrocke350m1 kernel: [ 2313] 0 2313 1008 160 6 3 0 0 flock
Feb 12 08:22:01 asrocke350m1 kernel: [ 2314] 0 2314 1183 726 7 3 0 0 updatedb.mlocat
Feb 12 08:22:01 asrocke350m1 kernel: [ 2780] 1000 2780 2559 1409 8 3 0 0 ssh
Feb 12 08:22:01 asrocke350m1 kernel: [ 2789] 1000 2789 1773 1264 7 3 0 0 bash
Feb 12 08:22:01 asrocke350m1 kernel: [ 2798] 1000 2798 208137 83634 291 3 0 0 firefox-esr
Feb 12 08:22:01 asrocke350m1 kernel: [ 2807] 1000 2807 1760 1253 7 3 0 0 bash
Feb 12 08:22:01 asrocke350m1 kernel: [ 2833] 1000 2833 3332 1638 11 3 0 0 gconfd-2
Feb 12 08:22:01 asrocke350m1 kernel: [ 2883] 1000 2883 2944 2168 8 3 0 0 vim
Feb 12 08:22:02 asrocke350m1 kernel: [ 2895] 1000 2895 1784 1327 7 3 0 0 bash
Feb 12 08:22:03 asrocke350m1 kernel: [ 4298] 1000 4298 3444 2800 10 3 0 0 debcheckout
Feb 12 08:22:03 asrocke350m1 kernel: [ 4302] 1000 4302 1898 998 7 3 0 0 git
Feb 12 08:22:03 asrocke350m1 kernel: [ 4303] 1000 4303 11054 6508 22 3 0 0 git-remote-http
Feb 12 08:22:04 asrocke350m1 kernel: [ 4306] 1000 4306 4232 983 8 3 0 0 git
Feb 12 08:22:04 asrocke350m1 kernel: [ 4309] 1000 4309 2621 1522 8 3 0 0 git
Feb 12 08:22:04 asrocke350m1 kernel: [ 4322] 0 4322 1408 829 6 3 0 0 inetd
Feb 12 08:22:05 asrocke350m1 kernel: Out of memory: Kill process 2798 (firefox-esr) score 27 or sacrifice child
Feb 12 08:22:05 asrocke350m1 kernel: Killed process 2798 (firefox-esr) total-vm:832548kB, anon-rss:248168kB, file-rss:86300kB, shmem-rss:68kB
Feb 12 08:22:06 asrocke350m1 kernel: perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
```

The sum of the RSS values is 312,260. According to the article [1], one
page is 4 KB in size. That make it less then 1.3 GB.

```
$ more /proc/meminfo # after OOM run
MemTotal:        7886032 kB
MemFree:         1613032 kB
MemAvailable:    4510132 kB
Buffers:          160536 kB
Cached:          3103908 kB
SwapCached:            0 kB
Active:          4004748 kB
Inactive:        1629624 kB
Active(anon):    2255340 kB
Inactive(anon):   158560 kB
Active(file):    1749408 kB
Inactive(file):  1471064 kB
Unevictable:         232 kB
Mlocked:             232 kB
HighTotal:       7065828 kB
HighFree:        1555764 kB
LowTotal:         820204 kB
LowFree:           57268 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Dirty:                12 kB
Writeback:             0 kB
AnonPages:       2370220 kB
Mapped:           419968 kB
Shmem:             43972 kB
Slab:             460304 kB
SReclaimable:     423320 kB
SUnreclaim:        36984 kB
KernelStack:        4224 kB
PageTables:        14580 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8137316 kB
Committed_AS:    5379296 kB
VmallocTotal:     122880 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       10232 kB
DirectMap2M:      901120 kB
```

So I wonder, why the OOM killer kicked in at all.

Hints and insight is appreciated.


Thanks,

Paul


[1] http://careers.directi.com/display/tu/Understanding+and+optimizing+Memory+utilization[unhandled content-type:application/pgp-signature]
\
 
 \ /
  Last update: 2017-02-12 15:26    [W:0.109 / U:0.396 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site