lkml.org 
[lkml]   [2019]   [Aug]   [2]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: Possible mem cgroup bug in kernels between 4.18.0 and 5.3-rc1.
Date


> On Aug 2, 2019, at 7:41 AM, Michal Hocko <mhocko@kernel.org> wrote:
>
> On Fri 02-08-19 07:18:17, Masoud Sharbiani wrote:
>>
>>
>>> On Aug 2, 2019, at 12:40 AM, Michal Hocko <mhocko@kernel.org> wrote:
>>>
>>> On Thu 01-08-19 11:04:14, Masoud Sharbiani wrote:
>>>> Hey folks,
>>>> I’ve come across an issue that affects most of 4.19, 4.20 and 5.2 linux-stable kernels that has only been fixed in 5.3-rc1.
>>>> It was introduced by
>>>>
>>>> 29ef680 memcg, oom: move out_of_memory back to the charge path
>>>
>>> This commit shouldn't really change the OOM behavior for your particular
>>> test case. It would have changed MAP_POPULATE behavior but your usage is
>>> triggering the standard page fault path. The only difference with
>>> 29ef680 is that the OOM killer is invoked during the charge path rather
>>> than on the way out of the page fault.
>>>
>>> Anyway, I tried to run your test case in a loop and leaker always ends
>>> up being killed as expected with 5.2. See the below oom report. There
>>> must be something else going on. How much swap do you have on your
>>> system?
>>
>> I do not have swap defined.
>
> OK, I have retested with swap disabled and again everything seems to be
> working as expected. The oom happens earlier because I do not have to
> wait for the swap to get full.
>

In my tests (with the script provided), it only loops 11 iterations before hanging, and uttering the soft lockup message.


> Which fs do you use to write the file that you mmap?

/dev/sda3 on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Part of the soft lockup path actually specifies that it is going through __xfs_filemap_fault():

[ 561.452933] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [leaker:3261]
[ 561.459904] Modules linked in: dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support dcdbas ipmi_ssif intel_powerc
lamp coretemp kvm_intel ses ipmi_si kvm enclosure scsi_transport_sas ipmi_devintf irqbypass pcspkr lpc_ich sg joydev ipmi_msghandler wmi acp
i_power_meter acpi_cpufreq xfs libcrc32c ata_generic sd_mod pata_acpi ata_piix libata megaraid_sas crc32c_intel serio_raw bnx2 bonding
[ 561.495979] CPU: 4 PID: 3261 Comm: leaker Tainted: G I L 5.3.0-rc2+ #10
[ 561.503704] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.4.0 07/23/2013
[ 561.511168] RIP: 0010:lruvec_lru_size+0x49/0xf0
[ 561.515687] Code: 41 89 ed b8 ff ff ff ff 45 31 f6 49 c1 e5 03 eb 19 48 63 d0 4c 89 e9 48 03 8b 88 00 00 00 48 8b 14 d5 60 a9 92 94 4c 03
34 11 <48> c7 c6 80 7c bf 94 89 c7 e8 89 d3 59 00 3b 05 27 eb ff 00 72 d1
[ 561.534418] RSP: 0018:ffffb5f886a3f640 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 561.541968] RAX: 0000000000000002 RBX: ffff96fca3bba400 RCX: 00003ef5d82059f0
[ 561.549085] RDX: ffff9702a7a40000 RSI: 0000000000000010 RDI: ffffffff94bf7c80
[ 561.556202] RBP: 0000000000000001 R08: 0000000000000000 R09: ffffffff94ae1c00
[ 561.563318] R10: ffff96fcc7802520 R11: 0000000000000000 R12: 0000000000000004
[ 561.570435] R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
[ 561.577553] FS: 00007f5522602740(0000) GS:ffff9702a7a80000(0000) knlGS:0000000000000000
[ 561.585623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 561.591352] CR2: 00007fba755f95b0 CR3: 0000000c646dc000 CR4: 00000000000006e0
[ 561.598468] Call Trace:
[ 561.600907] shrink_node_memcg+0xc8/0x790
[ 561.604905] ? shrink_slab+0x245/0x280
[ 561.608644] ? mem_cgroup_iter+0x10a/0x2c0
[ 561.612728] shrink_node+0xcd/0x490
[ 561.616208] do_try_to_free_pages+0xda/0x3a0
[ 561.620466] ? mem_cgroup_select_victim_node+0x43/0x2f0
[ 561.625678] try_to_free_mem_cgroup_pages+0xe7/0x1c0
[ 561.630629] try_charge+0x246/0x7a0
[ 561.634107] mem_cgroup_try_charge+0x6b/0x1e0
[ 561.638453] ? mem_cgroup_commit_charge+0x5a/0x110
[ 561.643231] __add_to_page_cache_locked+0x195/0x330
[ 561.648100] ? scan_shadow_nodes+0x30/0x30
[ 561.652184] add_to_page_cache_lru+0x39/0xa0
[ 561.656442] iomap_readpages_actor+0xf2/0x230
[ 561.660787] iomap_apply+0xa3/0x130
[ 561.664266] iomap_readpages+0x97/0x180
[ 561.668091] ? iomap_migrate_page+0xe0/0xe0
[ 561.672266] read_pages+0x57/0x180
[ 561.675657] __do_page_cache_readahead+0x1ac/0x1c0
[ 561.680436] ondemand_readahead+0x168/0x2a0
[ 561.684606] filemap_fault+0x30d/0x830
[ 561.688343] ? flush_tlb_func_common.isra.8+0x147/0x230
[ 561.693554] ? __mod_lruvec_state+0x40/0xe0
[ 561.697726] ? alloc_set_pte+0x4e6/0x5b0
[ 561.701669] __xfs_filemap_fault+0x61/0x190 [xfs]
[ 561.706361] __do_fault+0x38/0xb0
[ 561.709666] __handle_mm_fault+0xbee/0xe90
[ 561.713750] handle_mm_fault+0xe2/0x200
[ 561.717574] __do_page_fault+0x224/0x490
[ 561.721485] do_page_fault+0x31/0x120
[ 561.725137] page_fault+0x3e/0x50
[ 561.728439] RIP: 0033:0x400c5a
[ 561.731483] Code: 45 c0 48 89 c6 bf 77 0e 40 00 b8 00 00 00 00 e8 3c fb ff ff c7 45 dc 00 00 00 00 eb 36 8b 45 dc 48 63 d0 48 8b 45 c0 48
01 d0 <0f> b6 00 0f be c0 01 45 e8 8b 45 dc 25 ff 0f 00 00 85 c0 75 10 8b
[ 561.750214] RSP: 002b:00007fffba1d9450 EFLAGS: 00010206
[ 561.755426] RAX: 00007f550346b000 RBX: 0000000000000000 RCX: 000000000000001a
[ 561.762542] RDX: 0000000001c4c000 RSI: 000000007fffffe5 RDI: 0000000000000000
[ 561.769659] RBP: 00007fffba1da4a0 R08: 0000000000000000 R09: 00007f552206c20d
[ 561.776775] R10: 0000000000000002 R11: 0000000000000246 R12: 0000000000400850
[ 561.783892] R13: 00007fffba1da580 R14: 0000000000000000 R15: 0000000000000000


If I switch the backing file to a ext4 filesystem (separate hard drive), it OOMs.


If I switch the file used to /dev/zero, it OOMs:

Todal sum was 0. Loop count is 11
Buffer is @ 0x7f2b66c00000
./test-script-devzero.sh: line 16: 3561 Killed ./leaker -p 10240 -c 100000


> Or could you try to
> simplify your test even further? E.g. does everything work as expected
> when doing anonymous mmap rather than file backed one?

It also OOMs with MAP_ANON.

Hope that helps.
Masoud


> --
> Michal Hocko
> SUSE Labs

[unhandled content-type:application/pkcs7-signature]
\
 
 \ /
  Last update: 2019-08-02 20:04    [W:0.113 / U:0.352 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site