lkml.org 
[lkml]   [2014]   [Apr]   [2]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: VDSO pvclock may increase host cpu consumption, is this a problem?
    On Wed, Apr 2, 2014 at 3:05 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
    > On Tue, Apr 01, 2014 at 05:46:34PM -0700, Andy Lutomirski wrote:
    >> On Tue, Apr 1, 2014 at 5:29 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
    >> > On Tue, Apr 01, 2014 at 12:17:16PM -0700, Andy Lutomirski wrote:
    >> >> On Tue, Apr 1, 2014 at 11:01 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
    >> >> > On Mon, Mar 31, 2014 at 10:33:41PM -0700, Andy Lutomirski wrote:
    >> >> >> On Mar 31, 2014 8:45 PM, "Marcelo Tosatti" <mtosatti@redhat.com> wrote:
    >> >> >> >
    >> >> >> > On Mon, Mar 31, 2014 at 10:52:25AM -0700, Andy Lutomirski wrote:
    >> >> >> > > On 03/29/2014 01:47 AM, Zhanghailiang wrote:
    >> >> >> > > > Hi,
    >> >> >> > > > I found when Guest is idle, VDSO pvclock may increase host consumption.
    >> >> >> > > > We can calcutate as follow, Correct me if I am wrong.
    >> >> >> > > > (Host)250 * update_pvclock_gtod = 1500 * gettimeofday(Guest)
    >> >> >> > > > In Host, VDSO pvclock introduce a notifier chain, pvclock_gtod_chain in timekeeping.c. It consume nearly 900 cycles per call. So in consideration of 250 Hz, it may consume 225,000 cycles per second, even no VM is created.
    >> >> >> > > > In Guest, gettimeofday consumes 220 cycles per call with VDSO pvclock. If the no-kvmclock-vsyscall is configured, gettimeofday consumes 370 cycles per call. The feature decrease 150 cycles consumption per call.
    >> >> >> > > > When call gettimeofday 1500 times,it decrease 225,000 cycles,equal to the host consumption.
    >> >> >> > > > Both Host and Guest is linux-3.13.6.
    >> >> >> > > > So, whether the host cpu consumption is a problem?
    >> >> >> > >
    >> >> >> > > Does pvclock serve any real purpose on systems with fully-functional
    >> >> >> > > TSCs? The x86 guest implementation is awful, so it's about 2x slower
    >> >> >> > > than TSC. It could be improved a lot, but I'm not sure I understand why
    >> >> >> > > it exists in the first place.
    >> >> >> >
    >> >> >> > VM migration.
    >> >> >>
    >> >> >> Why does that need percpu stuff? Wouldn't it be sufficient to
    >> >> >> interrupt all CPUs (or at least all cpus running in userspace) on
    >> >> >> migration and update the normal timing data structures?
    >> >> >
    >> >> > Are you suggesting to allow interruption of the timekeeping code
    >> >> > at any time to update frequency information ?
    >> >>
    >> >> I'm not sure what you mean by "interruption of the timekeeping code".
    >> >> I'm suggesting sending an interrupt to the guest (via a virtio device,
    >> >> presumably) to tell it that it has been paused and resumed.
    >> >>
    >> >> This is probably worth getting John's input if you actually want to do
    >> >> this. I'm not about to :)
    >> >
    >> > Honestly, neither am i at the moment. But i'll think about it.
    >> >
    >> >> Is there any case in which the TSC is stable and the kvmclock data for
    >> >> different cpus is actually different?
    >> >
    >> > No. However, kvmclock_data.flags field is an interface for watchdog
    >> > unpause.
    >> >
    >> >> > Do you want to that as a special tsc clocksource driver ?
    >> >> >
    >> >> >> Even better: have the VM offer to invalidate the physical page
    >> >> >> containing the kernel's clock data on migration and interrupt one CPU.
    >> >> >> If another CPU races, it'll fault and wait for the guest kernel to
    >> >> >> update its timing.
    >> >> >
    >> >> > Perhaps that is a good idea.
    >> >> >
    >> >> >> Does the current kvmclock stuff track CLOCK_MONOTONIC and
    >> >> >> CLOCK_REALTIME separately?
    >> >> >
    >> >> > No. kvmclock counting is interrupted on vm pause (the "hw" clock does not
    >> >> > count during vm pause).
    >> >>
    >> >> Makes sense.
    >> >>
    >> >> >
    >> >> >> > Can you explain why you consider it so bad ? How you think it could be
    >> >> >> > improved ?
    >> >> >>
    >> >> >> The second rdtsc_barrier looks unnecessary. Even better, if rdtscp is
    >> >> >> available, then rdtscp can replace rdtsc_barrier, rdtsc, and the
    >> >> >> getcpu call.
    >> >> >>
    >> >> >> It would also be nice to avoid having two sets of rescalings of the timing data.
    >> >> >
    >> >> > Yep, probably good improvements, patches are welcome :-)
    >> >> >
    >> >>
    >> >> I may get to it at some point. No guarantees. I did just rewrite all
    >> >> the mapping-related code for every other x86 vdso timesource, so maybe
    >> >> I should try to add this to the pile. The fact that the data is a
    >> >> variable number of pages makes it messy, though, and since I don't
    >> >> understand why there's a separate structure for each CPU, I'm hesitant
    >> >> to change it too much.
    >> >>
    >> >> --Andy
    >> >
    >> > kvmclock.data? Because each VCPU can have different .flags fields for
    >> > example.
    >>
    >> It looks like the vdso kvmclock code only runs if
    >> PVCLOCK_TSC_STABLE_BIT is set, which in turn is only the case if the
    >> TSC is guaranteed to be monotonic across all CPUs. If we can rely on
    >> the fact that that bit will only be set if tsc_to_system_mul and
    >> tsc_shift are the same on all CPUs and that (system_time -
    >> (tsc_timestamp * mul) >> shift) is the same on all CPUs, then there
    >> should be no reason for the vdso to read the pvclock data for anything
    >> but CPU 0. That will make it a lot faster and simpler.
    >>
    >> Can we rely on that?
    >
    > In theory yes, but you would have to handle
    >
    > PVCLOCK_TSC_STABLE_BIT set -> PVCLOCK_TSC_STABLE_BIT not set
    >
    > Transition (and the other way around as well).

    Since !STABLE already results in a real syscall for clock_gettime and
    gettimeofday, I don't think this is a real hardship for the vdso.

    >
    >> I wonder what happens if the guest runs ntpd or otherwise uses
    >> adjtimex. Presumably it starts drifting relative to the host.
    >
    > It should use ntpd and adjtimex. KVMCLOCK is the "hw" clock,
    > the values returned by CLOCK_REALTIME and CLOCK_GETTIME are built
    > by the Linux guest timekeeping subsystem on top of the "hw" clock.
    >

    If the kernel can guarantee that, then the timing code gets faster,
    since the cyc2ns scale will be unity. Maybe this is worth a branch.

    Anyway, I'll try to find some time to improve this if/when hpa picks
    up my current series of vdso cleanups. I suspect that the overall
    effect will be a 30-40% speedup in clock_gettime along with a decent
    reduction of code complexity.

    --Andy


    \
     
     \ /
      Last update: 2014-04-03 01:01    [W:5.481 / U:0.232 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site