lkml.org 
[lkml]   [2021]   [Jun]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
Date
From: Mark Rutland <mark.rutland@arm.com> Sent: Tuesday, May 18, 2021 10:00 AM
>
> On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
> > >
> > > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > > > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > > > Add architecture specific definitions and functions needed
> > > > > > by the architecture independent Hyper-V clocksource driver.
> > > > > > Update the Hyper-V clocksource driver to be initialized
> > > > > > on ARM64.
> > > > >
> > > > > Previously we've said that for a clocksource we must use the architected
> > > > > counter, since that's necessary for things like the VDSO to work
> > > > > correctly and efficiently.
> > > > >
> > > > > Given that, I'm a bit confused that we're registering a per-cpu
> > > > > clocksource that is in part based on the architected counter. Likewise,
> > > > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > > > >
> > > > > Are the architected counter and timer reliable without this PV
> > > > > infrastructure? Why do we need to PV either of those?
> > > > >
> > > > > Thanks,
> > > > > Mark.
> > > >
> > > > For the clocksource, we have a requirement to live migrate VMs
> > > > between Hyper-V hosts running on hardware that may have different
> > > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > > > requirement). The Hyper-V virtualization does scaling to handle the
> > > > frequency difference. And yes, there's a tradeoff with vDSO not
> > > > working, though we have an out-of-tree vDSO implementation that
> > > > we can use when necessary.
> > >
> > > Just to be clear, the vDSO is *one example* of something that won't
> > > function correctly. More generally, because this undermines core
> > > architectural guarantees, it requires more invasive changes (e.g. we'd
> > > have to weaken the sanity checks, and not use the counter in things like
> > > kexec paths), impacts any architectural features tied to the generic
> > > timer/counter (e.g. the event stream, SPE and tracing, future features),
> > > and means that other SW (e.g. bootloaders and other EFI applications)
> > > are unlikley to function correctly in this environment.
> > >
> > > I am very much not keen on trying to PV this.
> > >
> > > What does the guest see when it reads CNTFRQ_EL0? Does this match the
> > > real HW value (and can this change over time)? Or is this entirely
> > > synthetic?
> > >
> > > What do the ACPI tables look like in the guest? Is there a GTDT table at
> > > all?
> > >
> > > How does the counter event stream behave?
> > >
> > > Are there other architectural features which Hyper-V does not implement
> > > for a guest?
> > >
> > > Is there anything else that may change across a migration? e.g. MIDR?
> > > MPIDR? Any of the ID registers?
> >
> > The ARMv8 architectural system counter and associated registers are visible
> > and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
> > instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
> > and a Linux guest on Hyper-V runs fine with this clocksource. Low level code
> > like bootloaders and EFI applications work normally.
>
> That's good to hear!
>
> One potential issue worth noting is that as those pieces of software are
> unlikely to handle counter frequency changes reliably, and so may not
> behave correctly if live-migrated.
>
> > The Hyper-V virtualization provides another Linux clocksource that is an
> > overlay on the arch counter and that provides time consistency across a live
> > migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
> > but the Hyper-V team believes they can make it functional. I have not
> > explored with them the live migration implications of things beyond time
> > consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
> >
> > Would a summary of your point be that live migration across hardware
> > with different arch counter frequencies is likely to not be possible with
> > 100% fidelity because of these other dependencies on the arch counter
> > frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
>
> Yes.
>
> In addition, there are a larger set of things necessarily exposed to VMs
> that mean that live migration isn't all that practical except betweenm
> identical machines (where the counter frequency should be identical),
> and the timer frequency might just be the canary in the coalmine. For
> example, the cache properties enumerated in CTR_EL0 cannot necessarily
> be emulated on another machine.
>
> > > > For clockevents, the only timer interrupt that Hyper-V provides
> > > > in a guest VM is its virtualized "STIMER" interrupt. There's no
> > > > virtualization of the ARM arch timer in the guest.
> > >
> > > I think that is rather unfortunate, given it's a core architectural
> > > feature. Is it just the interrupt that's missing? i.e. does all the
> > > PE-local functionality behave as the architecture requires?
> >
> > Right off the bat, I don't know about timer-related PE-local
> > functionality as it's not exercised in a Linux VM on Hyper-V that is
> > using STIMER-based clockevents. I'll explore with the Hyper-V
> > team. My impression is that enabling the ARM arch timer in a
> > guest VM is more work for Hyper-V than just wiring up an
> > interrupt.
>
> Thanks for chasing this up!
>

I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and
I'll resubmit the code so that arm_arch_timer.c provides the
standard arch_sys_counter clocksource. As noted previously, this just
works for a Hyper-V guest. The live migration discussion may come
back later after a deeper investigation by Hyper-V.

For clockevents, there's not a near term fix. It's more than just plumbing
an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
From their perspective there's also benefit in having a timer abstraction
that's independent of the architecture, and in the Linux guest, the STIMER
code is common across x86/x64 and ARM64. It follows the standard Linux
clockevents model, as it should. The code is already in use in out-of-tree
builds in the Linux VMs included in Windows 10 on ARM64 as part of the
so-called "Windows Subsystem for Linux".

So I'm hoping we can get this core support for ARM64 guests on Hyper-V
into upstream using the existing STIMER support. At some point, Hyper-V
will do the virtualization of the ARM64 arch timer, but we don't want to
have to stay out-of-tree until after that happens.

Thoughts?

Michael

\
 
 \ /
  Last update: 2021-06-08 17:36    [W:0.129 / U:0.232 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site