lkml.org 
[lkml]   [2010]   [Sep]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [KVM timekeeping 10/35] Fix deep C-state TSC desynchronization
Am 15.09.2010 01:40, Zachary Amsden wrote:
> On 09/14/2010 12:26 PM, Jan Kiszka wrote:
>> Am 14.09.2010 21:32, Zachary Amsden wrote:
>>
>>> On 09/14/2010 12:40 AM, Jan Kiszka wrote:
>>>
>>>> Am 14.09.2010 11:27, Avi Kivity wrote:
>>>>
>>>>
>>>>> On 09/14/2010 11:10 AM, Jan Kiszka wrote:
>>>>>
>>>>>
>>>>>> Am 20.08.2010 10:07, Zachary Amsden wrote:
>>>>>>
>>>>>>
>>>>>>> When CPUs with unstable TSCs enter deep C-state, TSC may stop
>>>>>>> running. This causes us to require resynchronization. Since
>>>>>>> we can't tell when this may potentially happen, we assume the
>>>>>>> worst by forcing re-compensation for it at every point the VCPU
>>>>>>> task is descheduled.
>>>>>>>
>>>>>>> Signed-off-by: Zachary Amsden<zamsden@redhat.com>
>>>>>>> ---
>>>>>>> arch/x86/kvm/x86.c | 2 +-
>>>>>>> 1 files changed, 1 insertions(+), 1 deletions(-)
>>>>>>>
>>>>>>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>>>>>>> index 7fc4a55..52b6c21 100644
>>>>>>> --- a/arch/x86/kvm/x86.c
>>>>>>> +++ b/arch/x86/kvm/x86.c
>>>>>>> @@ -1866,7 +1866,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu
>>>>>>> *vcpu, int cpu)
>>>>>>> }
>>>>>>>
>>>>>>> kvm_x86_ops->vcpu_load(vcpu, cpu);
>>>>>>> - if (unlikely(vcpu->cpu != cpu)) {
>>>>>>> + if (unlikely(vcpu->cpu != cpu) || check_tsc_unstable()) {
>>>>>>> /* Make sure TSC doesn't go backwards */
>>>>>>> s64 tsc_delta = !vcpu->arch.last_host_tsc ? 0 :
>>>>>>> native_read_tsc() - vcpu->arch.last_host_tsc;
>>>>>>>
>>>>>>>
>>>>>> For yet unknown reason, this commit breaks Linux guests here if they
>>>>>> are
>>>>>> started with only a single VCPU. They hang during boot, obviously no
>>>>>> longer receiving interrupts.
>>>>>>
>>>>>> I'm using kvm-kmod against a 2.6.34 host kernel, so this may be a
>>>>>> side
>>>>>> effect of the wrapping, though I cannot imagine how.
>>>>>>
>>>>>> Anyone any ideas?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Most likely, time went backwards, and some 'future - past' calculation
>>>>> resulted in a negative sleep value which was then interpreted as
>>>>> unsigned and resulted in a 2342525634 year sleep.
>>>>>
>>>>>
>>>> Looks like that's the case on first glance at the apic state.
>>>>
>>>>
>>> This compensation effectively nulls the delta between current and
>>> last TSC:
>>>
>>> if (unlikely(vcpu->cpu != cpu) || check_tsc_unstable()) {
>>> /* Make sure TSC doesn't go backwards */
>>> s64 tsc_delta = !vcpu->arch.last_host_tsc ? 0 :
>>> native_read_tsc() -
>>> vcpu->arch.last_host_tsc;
>>> if (tsc_delta< 0)
>>> mark_tsc_unstable("KVM discovered backwards
>>> TSC");
>>> if (check_tsc_unstable())
>>> kvm_x86_ops->adjust_tsc_offset(vcpu,
>>> -tsc_delta);
>>> kvm_migrate_timers(vcpu);
>>> vcpu->cpu = cpu;
>>>
>>> If TSC has advanced quite a bit due to a TSC jump during sleep(*), it
>>> will adjust the offset backwards to compensate; similarly, if it has
>>> gone backwards, it will advance the offset.
>>>
>>> In neither case should the visible TSC go backwards, assuming
>>> last_host_tsc is recorded properly, and so kvmclock should be similarly
>>> unaffected.
>>>
>>> Perhaps the guest is more intelligent than we hope, and is comparing two
>>> different clocks: kvmclock or TSC with the rate of PIT interrupts. This
>>> could result in negative arithmetic begin interpreted as unsigned. Are
>>> you using PIT interrupt reinjection on this guest or passing
>>> -no-kvm-pit-reinjection?
>>>
>>>
>>>>
>>>>
>>>>> Does your guest use kvmclock, tsc, or some other time source?
>>>>>
>>>>>
>>>> A kernel that has kvmclock support even hangs in SMP mode. The others
>>>> pick hpet or acpi_pm. TSC is considered unstable.
>>>>
>>>>
>>> SMP mode here has always and will always be unreliable. Are you running
>>> on an Intel or AMD CPU? The origin of this code comes from a workaround
>>> for (*) in vendor-specific code, and perhaps it is inappropriate for
>>> both.
>>>
>> I'm on a fairly new Intel i7 (M 620). And I accidentally rebooted my box
>> a few hours ago. Well, the issue is gone now...
>>
>> So I looked into the system logs and found this:
>>
>> [18446744053.434939] PM: resume of devices complete after 4379.595 msecs
>> [18446744053.457133] PM: Finishing wakeup.
>> [18446744053.457135] Restarting tasks ...
>> [ 0.000999] Marking TSC unstable due to KVM discovered backwards TSC
>> [270103.974668] done.
>>
>> From that point on the box was on hpet, including the time I did the
>> failing tests this morning. The kvm-kmod version loaded at this point
>> was based on kvm.git df549cfc.
>>
>> But my /proc/cpuinfo claims "constant_tsc", and Linux is generally happy
>> with using it as clock source. Does this tell you anything?
>>
>
> Yes, quite a bit.
>
> It's possible that marking the TSC unstable with an actively running VM
> causes a boundary condition that I had not accounted for. It's also
> possible that the clocksource switch triggered some bad behavior.

Suspend/resume (to RAM) is indeed triggering the tsc switch by KVM here.
This should be the first issue as the kernel itself has no problems with
recovering from suspend/resume /wrt tsc.

The next one is what happened to the guest running at that point. It was
a SUSE 11.3 32-bit image, using kvm-clock. After resume and host-side
clock switch it lost its timer ticks, likely due to some breakage of
kvm-clock.

And finally, I'm now in the original failure state again in which every
newly started Linux guest with kvm-clock support also suffers from stuck
timers. Linux kernel that lack kvm-clock run fine, e.g. on hpet
clocksource. Maybe this is just another symptom of what also cause the
second problem.

>
> This suggests two debugging techniques: I can manually switch the
> clocksource, and I can also load a module which does nothing other than
> mark the TSC unstable. Failing that, we can investigate PM suspend /
> resume for possible issues.
>
> I'll try this on my Intel boxes to see what happens.

Do you think kvm-kmod could contribute to this? As I said, I'm on a 34
kernel, namely SUSE's 2.6.34.4-0.1-desktop. Any feature missing in that
kernel latest KVM depends for proper tsc/kvm-clock handling? If you have
any concerns, I could try to run kvm.git natively later on.

Jan

[unhandled content-type:application/pgp-signature]
\
 
 \ /
  Last update: 2010-09-15 07:37    [W:0.064 / U:0.052 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site