Messages in this thread | | | Date | Tue, 17 Sep 2013 20:22:42 -0400 | From | Peter Hurley <> | Subject | Re: increased vmap_area_lock contentions on "n_tty: Move buffers into n_tty_data" |
| |
On 09/17/2013 07:22 PM, Fengguang Wu wrote: > On Tue, Sep 17, 2013 at 11:34:21AM -0400, Peter Hurley wrote: >> On 09/12/2013 09:09 PM, Fengguang Wu wrote: >>> On Fri, Sep 13, 2013 at 08:51:33AM +0800, Fengguang Wu wrote: >>>> Hi Peter, >>>> >>>> FYI, we noticed much increased vmap_area_lock contentions since this >>>> commit: >>>> >>>> commit 20bafb3d23d108bc0a896eb8b7c1501f4f649b77 >>>> Author: Peter Hurley <peter@hurleysoftware.com> >>>> Date: Sat Jun 15 10:21:19 2013 -0400 >>>> >>>> n_tty: Move buffers into n_tty_data >>>> >>>> Reduce pointer reloading and improve locality-of-reference; >>>> allocate read_buf and echo_buf within struct n_tty_data. >>> >>> Here are some comparison between this commit [o] with its parent commit [*]. >> >> Hi Fengguang,
Sorry for misspelling your name earlier. Fixed.
>> Can you give the particulars of the aim7 test runs below? >> I ask because I get _no_ added contention on the vmap_area_lock when I run >> these tests on a dual-socket xeon. >> >> What is the machine configuration(s)? >> Are you using the aim7 'multitask' test driver or your own custom driver? >> What is the load configuration (ie., constant, linearly increasing, convergence)? >> How many loads are you simulating? > > The aim7 tests are basically > > ( > echo $HOSTNAME > echo $workfile > > echo 1 > echo 2000 > echo 2 > echo 2000 > echo 1 > ) | ./multitask -t
Thanks for the profile. I ran the aim7 tests with these load parameters (2000!) and didn't have any significant contention with vmap_area_lock (162).
I had to run a subset of the aim7 tests (just those below) because I don't have anything fast enough to simulate 2000 loads on the entire workfile.shared testsuite.
>> lock_stat.vmap_area_lock.holdtime-total >> [...] >>> 489739.50 +978.5% 5281916.05 lkp-ne04/micro/aim7/shell_rtns_1 >>> 1601675.63 +906.7% 16123642.52 lkp-snb01/micro/aim7/exec_test >> [...] >>> 822461.02 +1585.0% 13858430.62 nhm-white/micro/aim7/exec_test >>> 9858.11 +2715.9% 277595.41 nhm-white/micro/aim7/fork_test >> [...] >>> 300.14 +2621.5% 8168.53 nhm-white/micro/aim7/misc_rtns_1 >>> 345479.21 +1624.5% 5957828.25 nhm-white/micro/aim7/shell_rtns_1 >> >> >> None of the tests below execute a code path that leads to get_vmalloc_info(). >> The only in-kernel user of get_vmalloc_info() is a sysfs read of /proc/meminfo, >> which none of the tests below perform. >> >> What is reading /proc/meminfo? > > Good point! That may explain it: I'm running a > > loop: > cat /proc/meminfo > sleep 1 > > in all the tests.
Yep. That's what's creating the contention -- while the aim7 test is creating ttys for each and every process (exec_test, shell_rtns_1, ...), the read of /proc/meminfo is contending with the allocations/frees of 2000 tty ldisc buffers.
Looking over vmalloc.c, the critical section footprint of the vmap_area_lock could definitely be reduced (even nearly eliminated), but that's a project for another day :)
Regards, Peter Hurley
>>> lock_stat.vmap_area_lock.contentions.get_vmalloc_info >>> >>> 8cb06c983822103da1cf 20bafb3d23d108bc0a89 >>> ------------------------ ------------------------ >>> 4952.40 +447.0% 27090.40 lkp-ne04/micro/aim7/shell_rtns_1 >>> 28410.80 +556.2% 186423.00 lkp-snb01/micro/aim7/exec_test >>> 8142.00 +615.4% 58247.33 nhm-white/micro/aim7/exec_test >>> 1386.00 +762.6% 11955.20 nhm-white/micro/aim7/shell_rtns_1 >>> 42891.20 +561.5% 283715.93 TOTAL lock_stat.vmap_area_lock.contentions.get_vmalloc_info
| |