lkml.org 
[lkml]   [2009]   [Jan]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [Bug #12465] KVM guests stalling on 2.6.28 (bisected)

    * Kevin Shanahan <kmshanah@ucwb.org.au> wrote:

    > On Tue, 2009-01-20 at 13:56 +0100, Ingo Molnar wrote:
    > > * Kevin Shanahan <kmshanah@ucwb.org.au> wrote:
    > > > > This suggests some sort of KVM-specific problem. Scheduler latencies
    > > > > in the seconds that occur under normal load situations are noticed and
    > > > > reported quickly - and there are no such open regressions currently.
    > > >
    > > > It at least suggests a problem with interaction between the scheduler
    > > > and kvm, otherwise reverting that scheduler patch wouldn't have made the
    > > > regression go away.
    > >
    > > the scheduler affects almost everything, so almost by definition a
    > > scheduler change can tickle a race or other timing bug in just about any
    > > code - and reverting that change in the scheduler can make the bug go
    > > away. But yes, it could also be a genuine scheduler bug - that is always a
    > > possibility.
    >
    > Okay, I understand.
    >
    > > Could you please run a cfs-debug-info.sh session on a CONFIG_SCHED_DEBUG=y
    > > and CONFIG_SCHEDSTATS=y kernel, while you are experiencing those
    > > latencies:
    > >
    > > http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
    > >
    > > and post that (relatively large) somewhere, or send it as a reply after
    > > bzip2 -9 compressing it? It will include a lot of information about the
    > > delays your tasks are experiencing.
    >
    > Running it while the problem is occuring will be tricky, as it only
    > lasts for a few seconds at a time. Is it going to be useful at all to
    > just see those statistics if the system is running normally?
    >
    > I might need to modify the script a little. Am I right that everything
    > above "gathering statistics..." is pretty much static information?

    Correct.

    > I could run top, vmstat and cat /proc/sched_debug in a loop until the
    > problem occurs and then trim it. Something like:
    >
    > while true; do
    > date >> $FILE
    > echo "-- top: --" >> $FILE
    > top -H -c -b -d 1 -n 0.5 >> $FILE 2>/dev/null
    > echo "-- vmstat: --" >> $FILE
    > vmstat >> $FILE 2>/dev/null
    > echo "-- sched_debug #$i: --" >> $FILE
    > cat /proc/sched_debug >> $FILE 2>/dev/null
    > done
    >
    > That should take a snapshot every half second or so.

    Yeah, that would be lovely. You dont even have to trim it much - just give
    us a timestamp to look at for the delay incident. You might also want to
    start the kvm session while the script is already running - that way we'll
    get fresh statistics and see the whole thing.

    Ingo


    \
     
     \ /
      Last update: 2009-01-20 15:29    [W:0.024 / U:149.776 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site