lkml.org 
[lkml]   [2009]   [Mar]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [RFC PATCH 00/19] Cleanup and optimise the page allocator V2
    From
    Date
    On Mon, 2009-03-02 at 11:21 +0000, Mel Gorman wrote:
    > (Added Ingo as a second scheduler guy as there are queries on tg_shares_up)
    >
    > On Fri, Feb 27, 2009 at 04:44:43PM +0800, Lin Ming wrote:
    > > On Thu, 2009-02-26 at 19:22 +0800, Mel Gorman wrote:
    > > > In that case, Lin, could I also get the profiles for UDP-U-4K please so I
    > > > can see how time is being spent and why it might have gotten worse?
    > >
    > > I have done the profiling (oltp and UDP-U-4K) with and without your v2
    > > patches applied to 2.6.29-rc6.
    > > I also enabled CONFIG_DEBUG_INFO so you can translate address to source
    > > line with addr2line.
    > >
    > > You can download the oprofile data and vmlinux from below link,
    > > http://www.filefactory.com/file/af2330b/
    > >
    >
    > Perfect, thanks a lot for profiling this. It is a big help in figuring out
    > how the allocator is actually being used for your workloads.
    >
    > The OLTP results had the following things to say about the page allocator.
    In case we might mislead you guys, I want to clarify that here OLTP is
    sysbench (oltp)+mysql, not the famous OLTP which needs lots of disks and big
    memory.

    Ma Chinang, another Intel guy, does work on the famous OLTP running.

    >
    > Samples in the free path
    > vanilla: 6207
    > mg-v2: 4911
    > Samples in the allocation path
    > vanilla 19948
    > mg-v2: 14238
    >
    > This is based on glancing at the following graphs and not counting the VM
    > counters as it can't be determined which samples are due to the allocator
    > and which are due to the rest of the VM accounting.
    >
    > http://www.csn.ul.ie/~mel/postings/lin-20090228/free_pages-vanilla-oltp.png
    > http://www.csn.ul.ie/~mel/postings/lin-20090228/free_pages-mgv2-oltp.png
    >
    > So the path costs are reduced in both cases. Whatever caused the regression
    > there doesn't appear to be in time spent in the allocator but due to
    > something else I haven't imagined yet. Other oddness
    >
    > o According to the profile, something like 45% of time is spent entering
    > the __alloc_pages_nodemask() function. Function entry costs but not
    > that much. Another significant part appears to be in checking a simple
    > mask. That doesn't make much sense to me so I don't know what to do with
    > that information yet.
    >
    > o In get_page_from_freelist(), 9% of the time is spent deleting a page
    > from the freelist.
    >
    > Neither of these make sense, we're not spending time where I would expect
    > to at all. One of two things are happening. Something like cache misses or
    > bounces are dominating for some reason that is specific to this machine. Cache
    > misses are one possibility that I'll check out. The other is that the sample
    > rate is too low and the profile counts are hence misleading.
    >
    > Question 1: Would it be possible to increase the sample rate and track cache
    > misses as well please?
    I will try to capture cache miss with oprofile.

    >
    > Another interesting fact is that we are spending about 15% of the overall
    > time is spent in tg_shares_up() for both kernels but the vanilla kernel
    > recorded 977348 samples and the patched kernel recorded 514576 samples. We
    > are spending less time in the kernel and it's not obvious why or if that is
    > a good thing or not. You'd think less time in kernel is good but it might
    > mean we are doing less work overall.
    >
    > Total aside from the page allocator, I checked what we were doing
    > in tg_shares_up where the vast amount of time is being spent. This has
    > something to do with CONFIG_FAIR_GROUP_SCHED.
    >
    > Question 2: Scheduler guys, can you think of what it means to be spending
    > less time in tg_shares_up please?
    >
    > I don't know enough of how it works to guess why we are in there. FWIW,
    > we are appear to be spending the most time in the following lines
    >
    > weight = tg->cfs_rq[i]->load.weight;
    > if (!weight)
    > weight = NICE_0_LOAD;
    >
    > tg->cfs_rq[i]->rq_weight = weight;
    > rq_weight += weight;
    > shares += tg->cfs_rq[i]->shares;
    >
    > So.... cfs_rq is SMP aligned, but we iterate though it with for_each_cpu()
    > and we're writing to it. How often is this function run by multiple CPUs? If
    > the answer is "lots", does that not mean we are cache line bouncing in
    > here like mad? Another crazy amount of time is spent accessing tg->se when
    > validating. Basically, any access of the task_group appears to incur huge
    > costs and cache line bounces would be the obvious explanation.
    FAIR_GROUP_SCHED is a feature to support configurable cpu weight for different users.
    We did find it takes lots of time to check/update the share weight which might create
    lots of cache ping-pang. With sysbench(oltp)+mysql, that becomes more severe because
    mysql runs as user mysql and sysbench runs as another regular user. When starting
    the testing with 1 thread in command line, there are 2 mysql threads and 1 sysbench
    thread are proactive.

    >
    > More stupid poking around. We appear to update these share things on each
    > fork().
    >
    > Question 3: Scheduler guys, If the database or clients being used for OLTP is
    > fork-based instead of thread-based, then we are going to be balancing a lot,
    > right? What does that mean, how can it be avoided?
    >
    > Question 4: Lin, this is unrelated to the page allocator but do you know
    > what the performance difference between vanilla-with-group-sched and
    > vanilla-without-group-sched is?
    When FAIR_GROUP_SCHED appeared in kernel at the first time, we did many such testing.
    There is another thread to discuss it at http://lkml.org/lkml/2008/9/10/214.

    set sched_shares_ratelimit to a large value could reduce the regression.

    Scheduler guys keep improving it.

    >
    > The UDP results are screwy as the profiles are not matching up to the
    > images. For example
    Mostly, it's caused by not cleaning up old oprofile data when starting
    new sampling.

    I will retry.

    >
    > oltp.oprofile.2.6.29-rc6: ffffffff802808a0 11022 0.1727 get_page_from_freelist
    > oltp.oprofile.2.6.29-rc6-mg-v2: ffffffff80280610 7958 0.2403 get_page_from_freelist
    > UDP-U-4K.oprofile.2.6.29-rc6: ffffffff802808a0 29914 1.2866 get_page_from_freelist
    > UDP-U-4K.oprofile.2.6.29-rc6-mg-v2: ffffffff802808a0 28153 1.1708 get_page_from_freelist
    >
    > Look at the addresses. UDP-U-4K.oprofile.2.6.29-rc6-mg-v2 has the address
    > for UDP-U-4K.oprofile.2.6.29-rc6 so I have no idea what I'm looking at here
    > for the patched kernel :(.
    >
    > Question 5: Lin, would it be possible to get whatever script you use for
    > running netperf so I can try reproducing it?
    Below is a simple script. As for formal testing, we add parameter "-i 50,3 -I" 99,5"
    to get a more stable result.

    PROG_DIR=/home/ymzhang/test/netperf/src
    taskset -c 0 ${PROG_DIR}/netserver
    sleep 2
    taskset -c 7 ${PROG_DIR}/netperf -t UDP_STREAM -l 60 -H 127.0.0.1 -- -P 15895 12391 -s 32768 -S 32768 -m 4096
    killall netserver


    Basically, we start 1 client and bind client/server to different physical cpu.

    >
    > Going by the vanilla kernel, a *large* amount of time is spent doing
    > high-order allocations. Over 25% of the cost of buffered_rmqueue() is in
    > the branch dealing with high-order allocations. Does UDP-U-4K mean that 8K
    > pages are required for the packets? That means high-order allocations and
    > high contention on the zone-list. That is bad obviously and has implications
    > for the SLUB-passthru patch because whether 8K allocations are handled by
    > SL*B or the page allocator has a big impact on locking.
    >
    > Next, a little over 50% of the cost get_page_from_freelist() is being spent
    > acquiring the zone spinlock. The implication is that the SL*B allocators
    > passing in order-1 allocations to the page allocator are currently going to
    > hit scalability problems in a big way. The solution may be to extend the
    > per-cpu allocator to handle magazines up to PAGE_ALLOC_COSTLY_ORDER. I'll
    > check it out.
    >

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2009-03-04 03:07    [W:0.034 / U:7.292 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site