lkml.org 
[lkml]   [2012]   [Oct]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH RFC 1/2] kvm: Handle undercommitted guest case in PLE handler
On 10/04/2012 06:11 PM, Avi Kivity wrote:
> On 10/04/2012 12:49 PM, Raghavendra K T wrote:
>> On 10/03/2012 10:35 PM, Avi Kivity wrote:
>>> On 10/03/2012 02:22 PM, Raghavendra K T wrote:
>>>>> So I think it's worth trying again with ple_window of 20000-40000.
>>>>>
>>>>
>>>> Hi Avi,
>>>>
>>>> I ran different benchmarks increasing ple_window, and results does not
>>>> seem to be encouraging for increasing ple_window.
>>>
>>> Thanks for testing! Comments below.
>>>
>>>> Results:
>>>> 16 core PLE machine with 16 vcpu guest.
>>>>
>>>> base kernel = 3.6-rc5 + ple handler optimization patch
>>>> base_pleopt_8k = base kernel + ple window = 8k
>>>> base_pleopt_16k = base kernel + ple window = 16k
>>>> base_pleopt_32k = base kernel + ple window = 32k
>>>>
>>>>
>>>> Percentage improvements of benchmarks w.r.t base_pleopt with
>>>> ple_window = 4096
>>>>
>>>> base_pleopt_8k base_pleopt_16k base_pleopt_32k
>>>> -----------------------------------------------------------------
>>>>
>>>> kernbench_1x -5.54915 -15.94529 -44.31562
>>>> kernbench_2x -7.89399 -17.75039 -37.73498
>>>
>>> So, 44% degradation even with no overcommit? That's surprising.
>>
>> Yes. Kernbench was run with #threads = #vcpu * 2 as usual. Is it
>> spending 8 times the original ple_window cycles for 16 vcpus
>> significant?
>
> A PLE exit when not overcommitted cannot do any good, it is better to
> spin in the guest rather that look for candidates on the host. In fact
> when we benchmark we often disable PLE completely.
>
>>
>>>
>>>> I also got perf top output to analyse the difference. Difference comes
>>>> because of flushtlb (and also spinlock).
>>>
>>> That's in the guest, yes?
>>
>> Yes. Perf is in guest.
>>
>>>
>>>>
>>>> Ebizzy run for 4k ple_window
>>>> - 87.20% [kernel] [k] arch_local_irq_restore
>>>> - arch_local_irq_restore
>>>> - 100.00% _raw_spin_unlock_irqrestore
>>>> + 52.89% release_pages
>>>> + 47.10% pagevec_lru_move_fn
>>>> - 5.71% [kernel] [k] arch_local_irq_restore
>>>> - arch_local_irq_restore
>>>> + 86.03% default_send_IPI_mask_allbutself_phys
>>>> + 13.96% default_send_IPI_mask_sequence_phys
>>>> - 3.10% [kernel] [k] smp_call_function_many
>>>> smp_call_function_many
>>>>
>>>>
>>>> Ebizzy run for 32k ple_window
>>>>
>>>> - 91.40% [kernel] [k] arch_local_irq_restore
>>>> - arch_local_irq_restore
>>>> - 100.00% _raw_spin_unlock_irqrestore
>>>> + 53.13% release_pages
>>>> + 46.86% pagevec_lru_move_fn
>>>> - 4.38% [kernel] [k] smp_call_function_many
>>>> smp_call_function_many
>>>> - 2.51% [kernel] [k] arch_local_irq_restore
>>>> - arch_local_irq_restore
>>>> + 90.76% default_send_IPI_mask_allbutself_phys
>>>> + 9.24% default_send_IPI_mask_sequence_phys
>>>>
>>>
>>> Both the 4k and the 32k results are crazy. Why is
>>> arch_local_irq_restore() so prominent? Do you have a very high
>>> interrupt rate in the guest?
>>
>> How to measure if I have high interrupt rate in guest?
>> From /proc/interrupt numbers I am not able to judge :(
>
> 'vmstat 1'
>

Thanks you. 'll save this. Apart from in,cs I think r: The number of
processes waiting for run time, would be useful for me in vmstat.

>>
>> I went back and got the results on a 32 core machine with 32 vcpu guest.
>> Strangely, I got result supporting the claim that increasing ple_window
>> helps for non-overcommitted scenario.
>>
>> 32 core 32 vcpu guest 1x scenarios.
>>
>> ple_gap = 0
>> kernbench: Elapsed Time 38.61
>> ebizzy: 7463 records/s
>>
>> ple_window = 4k
>> kernbench: Elapsed Time 43.5067
>> ebizzy: 2528 records/s
>>
>> ple_window = 32k
>> kernebench : Elapsed Time 39.4133
>> ebizzy: 7196 records/s
>
> So maybe something was wrong with the first measurement.

May be I was not clear. The first time I had run on x240 (sandybridge)
16 core cpu,

Then ran on 32 core x3850 to confirm the perf top results.
But yes both had

[ 0.018997] Performance Events: Broken PMU hardware detected, using
software events only.

problem as rightly pointed by you and PeterZ.

after -cpu host, I see that is fixed on x240,

[ 0.017997] Performance Events: 16-deep LBR, SandyBridge events,
Intel PMU driver.
[ 0.018868] NMI watchdog: enabled on all CPUs, permanently consumes
one hw-PMU counter.

So I 'll try it on x240 again.

( Some how mx3850 -cpu host resulted in
[ 0.026995] Performance Events: unsupported p6 CPU model 26 no PMU
driver, software events only.
I think qemu needs some fix as pointed in
http://www.mail-archive.com/kvm@vger.kernel.org/msg55836.html


>
>>
>>
>> perf top for ebizzy for above:
>> ple_gap = 0
>> - 84.74% [kernel] [k] arch_local_irq_restore
>> - arch_local_irq_restore
>> - 100.00% _raw_spin_unlock_irqrestore
>> + 50.96% release_pages
>> + 49.02% pagevec_lru_move_fn
>> - 6.57% [kernel] [k] arch_local_irq_restore
>> - arch_local_irq_restore
>> + 92.54% default_send_IPI_mask_allbutself_phys
>> + 7.46% default_send_IPI_mask_sequence_phys
>> - 1.54% [kernel] [k] smp_call_function_many
>> smp_call_function_many
>
> Again the numbers are ridiculously high for arch_local_irq_restore.
> Maybe there's a bad perf/kvm interaction when we're injecting an
> interrupt, I can't believe we're spending 84% of the time running the
> popf instruction.
>
>>
>> ple_window = 32k
>> - 84.47% [kernel] [k] arch_local_irq_restore
>> + arch_local_irq_restore
>> - 6.46% [kernel] [k] arch_local_irq_restore
>> - arch_local_irq_restore
>> + 93.51% default_send_IPI_mask_allbutself_phys
>> + 6.49% default_send_IPI_mask_sequence_phys
>> - 1.80% [kernel] [k] smp_call_function_many
>> - smp_call_function_many
>> + 99.98% native_flush_tlb_others
>>
>>
>> ple_window = 4k
>> - 91.35% [kernel] [k] arch_local_irq_restore
>> - arch_local_irq_restore
>> - 100.00% _raw_spin_unlock_irqrestore
>> + 53.19% release_pages
>> + 46.81% pagevec_lru_move_fn
>> - 3.90% [kernel] [k] smp_call_function_many
>> smp_call_function_many
>> - 2.94% [kernel] [k] arch_local_irq_restore
>> - arch_local_irq_restore
>> + 93.12% default_send_IPI_mask_allbutself_phys
>> + 6.88% default_send_IPI_mask_sequence_phys
>>
>> Let me know if I can try something here..
>> /me confused :(
>>
>
> I'm even more confused. Please try 'perf kvm' from the host, it does
> fewer dirty tricks with the PMU and so may be more accurate.
>

I will try with host perf kvm this time..



\
 
 \ /
  Last update: 2012-10-05 11:41    [W:0.249 / U:2.528 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site