Messages in this thread | | | From | "Rafael J. Wysocki" <> | Subject | Re: [PATCH 0/6] PM / Hibernate: Memory bitmap scalability improvements | Date | Sat, 19 Jul 2014 00:05:02 +0200 |
| |
On Friday, July 18, 2014 01:57:17 PM Joerg Roedel wrote: > Hi, > > here is a patch set to improve the scalability of the memory > bitmap implementation used for hibernation. The current > implementation does not scale well to machines with several > TB of memory. A resume on those machines may cause soft > lockups to be reported. > > These patches improve the data structure by adding a radix > tree to the linked list structure to improve random access > performance from O(n) to O(log_b(n)), where b depends on the > architecture (b=512 on amd64, 1024 in i386). > > A test on a 12TB machine showed an improvement in resume > time from 76s with the old implementation to 2.4s with the > radix tree and the improved swsusp_free function. See below > for details of this test. > > Patches 1-3 that add the radix tree while keeping the > existing memory bitmap implementation in place and add code > to compare the results between both implementations. This > was used during development to make sure both data > structures return the same results. > > Patch 4 re-implements the swsusp_free() function to not > iterate over all pfns but only over the bits set in the > bitmaps. This showed to scale better on large memory > machines. > > Patch 5 removes the old memory bitmap implementation now > that the radix tree is in place and working correctly. > > The last patch adds touching the soft lockup watchdog in > rtree_next_node. This is necessary because the worst case > performance (all bits set in the forbidden_pages_map and > free_pages_map) is the same as with the old implementation > and may still cause soft lockups. Patch 6 avoids this. > > The code was tested in 32 and 64 bit x86 and showed no > issues there. > > Below is an example test that shows the performance > improvement on a 12TB machine. First the test with the old > memory bitmap: > > # time perf record /usr/sbin/resume $sdev > resume: libgcrypt version: 1.5.0 > [ perf record: Woken up 12 times to write data ] > [ perf record: Captured and wrote 2.882 MB perf.data (~125898 samples) ] > > real 1m16.043s > user 0m0.016s > sys 0m0.312s > # perf report --stdio |head -50 > # Events: 75K cycles > # > # Overhead Command Shared Object > Symbol > # ........ ....... .................... > ........................................ > # > 56.16% resume [kernel.kallsyms] [k] memory_bm_test_bit > 19.35% resume [kernel.kallsyms] [k] swsusp_free > 14.90% resume [kernel.kallsyms] [k] memory_bm_find_bit > 7.28% resume [kernel.kallsyms] [k] swsusp_page_is_forbidden > > And here is the same test on the same machine with these > patches applied: > > # time perf record /usr/sbin/resume $sdev > resume: libgcrypt version: 1.5.0 > [ perf record: Woken up 1 times to write data ] > [ perf record: Captured and wrote 0.039 MB perf.data (~1716 samples) ] > > real 0m2.376s > user 0m0.020s > sys 0m0.408s > > # perf report --stdio |head -50 > # Events: 762 cycles > # > # Overhead Command Shared Object Symbol > # ........ ....... ................. ......................... > # > 34.78% resume [kernel.kallsyms] [k] find_next_bit > 27.03% resume [kernel.kallsyms] [k] clear_page_c_e > 9.70% resume [kernel.kallsyms] [k] mark_nosave_pages > 3.92% resume [kernel.kallsyms] [k] alloc_rtree_node > 2.38% resume [kernel.kallsyms] [k] get_image_page > > As can be seen on these results these patches improve the > scalability significantly. Please review, any comments > appreciated.
Looks good.
How much testing did that receive?
Rafael
| |