Messages in this thread | | | Date | Tue, 12 Jul 2016 10:27:37 +0200 | From | Matthias Dahl <> | Subject | Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage) |
| |
Hello,
I posted this issue already on linux-mm, linux-kernel and dm-devel a few days ago and after further investigation it seems like that this issue is somehow related to the fact that I am using an Intel Rapid Storage RAID10, so I am summarizing everything again in this mail and include linux-raid in my post. Sorry for the noise... :(
I am currently setting up a new machine (since my old one broke down) and I ran into a lot of " Unable to allocate memory on node -1" warnings while using dm-crypt. I have attached as much of the full log as I could recover.
The encrypted device is sitting on a RAID10 (software raid, Intel Rapid Storage). I am currently limited to testing via Linux live images since the machine is not yet properly setup but I did my tests across several of those.
Steps to reproduce are:
1) cryptsetup -s 512 -d /dev/urandom -c aes-xts-plain64 open --type plain /dev/md126p5 test-device
2) dd if=/dev/zero of=/dev/mapper/test-device status=progress bs=512K
While running and monitoring the memory usage with free, it can be seen that the used memory increases rapidly and after just a few seconds, the system is out of memory and page allocation failures start to be issued as well as the OOM killer gets involved.
I have also seen this behavior with mkfs.ext4 being used on the same device -- at least with 1.43.1.
Using direct i/o will work fine and not cause any issue. Also if dm-crypt is out of the picture, the problem does also not occur.
I did further tests:
1) dd block size has no influence on the issue whatsoever 2) using dm-crypt on an image located on an ext2 on the RAID10 works fine 3) using an external (connected through USB3) hd with two partitions and using either a RAID1 or RAID10 on it via Linux s/w RAID with dm-crypt on-top, does also work fine
But as soon as I use dm-crypt on the Intel Rapid Storage RAID10, the issue is 100% reproducible.
I tested all of this on a Fedora Rawhide Live Image as I currently still am in the process of setting the new machine up. Those images are available here to download:
download.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/x86_64/iso/
The machine itself has 32 GiB of RAM (plenty), no swap (live image) and is a 6700k on a Z170 chipset. The kernel is the default provided with the live image... right now that is a very recent git after 4.7.0rc6 but before rc7. But the issue also shows on 4.4.8 and 4.5.5.
The stripe size of the RAID10 is 64k, if that matters.
I am now pretty much out of ideas what else to test and where the problem could stem from. Suffice to say that this has impacted my trust in this particular setup. I hope I can help to find the cause of this.
If there is anything I can do to help, please let me know.
Also, since I am not subscribed to the lists right now (I have to make due with a crappy WebMail interface until everything is setup), please cc' me accordingly. Thanks a lot.
With Kind Regards from Germany, Matthias
-- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administrationPersonalities : [raid10] md126 : active raid10 sda[3] sdb[2] sdc[1] sdd[0] 3907023872 blocks super external:/md127/0 64K chunks 2 near-copies [4/4] [UUUU] md127 : inactive sdc[3](S) sdb[2](S) sda[1](S) sdd[0](S) 10064 blocks super external:imsm unused devices: <none> nr_free_pages 7943696 nr_alloc_batch 5873 nr_inactive_anon 296 nr_active_anon 105347 nr_inactive_file 29921 nr_active_file 56980 nr_unevictable 13140 nr_mlock 2716 nr_anon_pages 107204 nr_mapped 27670 nr_file_pages 98500 nr_dirty 14 nr_writeback 0 nr_slab_reclaimable 8887 nr_slab_unreclaimable 16975 nr_page_table_pages 7137 nr_kernel_stack 490 nr_unstable 0 nr_bounce 0 nr_vmscan_write 9828326 nr_vmscan_immediate_reclaim 67360474 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 593 nr_dirtied 506466654 nr_written 506466595 nr_pages_scanned 0 numa_hit 5258670960 numa_miss 0 numa_foreign 0 numa_interleave 38217 numa_local 5258670960 numa_other 0 workingset_refault 336993 workingset_activate 61553 workingset_nodereclaim 7919435 nr_anon_transparent_hugepages 0 nr_free_cma 0 nr_dirty_threshold 1592267 nr_dirty_background_threshold 795161 pgpgin 10489537 pgpgout 2025884337 pswpin 0 pswpout 0 pgalloc_dma 684558 pgalloc_dma32 328009673 pgalloc_normal 5767713958 pgalloc_movable 0 pgfree 6106436104 pgactivate 813221 pgdeactivate 1284082 pgfault 1653795 pgmajfault 46351 pglazyfreed 0 pgrefill_dma 0 pgrefill_dma32 66114 pgrefill_normal 1407169 pgrefill_movable 0 pgsteal_kswapd_dma 0 pgsteal_kswapd_dma32 22181873 pgsteal_kswapd_normal 425875886 pgsteal_kswapd_movable 0 pgsteal_direct_dma 0 pgsteal_direct_dma32 10723905 pgsteal_direct_normal 45330060 pgsteal_direct_movable 0 pgscan_kswapd_dma 0 pgscan_kswapd_dma32 32470709 pgscan_kswapd_normal 758168190 pgscan_kswapd_movable 0 pgscan_direct_dma 0 pgscan_direct_dma32 55064390 pgscan_direct_normal 449388285 pgscan_direct_movable 0 pgscan_direct_throttle 16 zone_reclaim_failed 0 pginodesteal 329 slabs_scanned 75784518 kswapd_inodesteal 3324 kswapd_low_wmark_hit_quickly 18086579 kswapd_high_wmark_hit_quickly 562 pageoutrun 18100603 allocstall 739928 pgrotated 357590082 drop_pagecache 0 drop_slab 0 numa_pte_updates 0 numa_huge_pte_updates 0 numa_hint_faults 0 numa_hint_faults_local 0 numa_pages_migrated 0 pgmigrate_success 562476 pgmigrate_fail 34076511 compact_migrate_scanned 390290706 compact_free_scanned 17609026156 compact_isolated 37387419 compact_stall 17 compact_fail 10 compact_success 7 compact_daemon_wake 3013752 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 69728 unevictable_pgs_scanned 0 unevictable_pgs_rescued 57566 unevictable_pgs_mlocked 62928 unevictable_pgs_munlocked 59182 unevictable_pgs_cleared 18 unevictable_pgs_stranded 18 thp_fault_alloc 0 thp_fault_fallback 0 thp_collapse_alloc 0 thp_collapse_alloc_failed 0 thp_split_page 0 thp_split_page_failed 0 thp_deferred_split_page 0 thp_split_pmd 0 thp_zero_page_alloc 0 thp_zero_page_alloc_failed 0 balloon_inflate 0 balloon_deflate 0 balloon_migrate 0 [unhandled content-type:application/x-gzip][unhandled content-type:application/x-gzip][unhandled content-type:application/x-gzip] | |