lkml.org 
[lkml]   [2009]   [Mar]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 0/5] [GIT PULL] updates for tip/tracing/ftrace
    On Sat, Mar 21, 2009 at 09:09:56PM +0100, Frederic Weisbecker wrote:
    > On Sat, Mar 21, 2009 at 07:18:58PM +0100, Ingo Molnar wrote:
    > >
    > > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
    > >
    > > > > [<ffffffff8020c79d>] return_to_handler+0x0/0x73
    > > > > [<ffffffff8029ea13>] rcu_pending+0x2c/0x5e
    > > > > [<ffffffff8020c79d>] return_to_handler+0x0/0x73
    > > > > [<ffffffff8026abef>] update_process_times+0x3c/0x77
    > > > > [<ffffffff8020c79d>] return_to_handler+0x0/0x73
    > > > > [<ffffffff802875dd>] tick_periodic+0x6e/0x70
    > > >
    > > >
    > > > Still hanging in the timer interrupt.
    > > > I guess it makes the timer interrupt servicing too slow and then
    > > > once it is serviced, another one is raised.
    > > >
    > > > But the cause is perhaps more complex
    > > >
    > > > I think you have had too much hanging of this type. I'm preparing
    > > > a fix that checks periodically if the function graph tracer is
    > > > spending too much time in an interrupt.
    > > >
    > > > I guess I could count the number of function executed between the
    > > > irq entry and its exit.
    > > >
    > > > That's the best: if we are hanging in an interrupt, it could be
    > > > whatever interrupt and the jiffies could not be progressing so I
    > > > can't rely on time but only on number of functions executed.
    > > >
    > > > May be 10000 calls is a good threshold before killing the function
    > > > graph inside an interrupt?
    > >
    > > i think the problem isnt even the IRQ handler - but the fact that
    > > the (timer) irq handler gets re-triggered - so all we do is
    > > processing timer IRQs.
    > >
    > > Your patch would detect a timer IRQ hanging - but it would not
    > > detect the 'system makes no progress because there's always anoter
    > > pending timer IRQ to execute' situation.
    >
    >
    > Ah, you're right.
    >
    >
    > > So i think we need a "function trace watchdog" - which kills the
    > > tracer if we do more than 100,000,000 entries since we started the
    > > self-test, or so.
    > >
    > > Ingo
    >
    >
    > The problem is that it can happen also on other contexts than selftests.
    > For example with ftrace=function_graph or by simply enabling the tracer
    > later.
    >
    > Sometimes it can happen during the selftests, sometimes it's only
    > revealed by manually enabling it. I just remind another hang that
    > you reported earlier and which I half-solved by fixing a pointless
    > softirq call...
    >

    Well, ok let's do that, it will be a first and good stage on debugging
    the graph hangs.
    I will write this selftest watchdog and ftrace_dump() once we reach
    100,000,000 entries.
    It will be very helpful to know what really happens and what can be
    optimized in this area.

    Concerning this ftrace_dump(), I will tune it to let us decide if we want
    to kill all tracing or not. For example, in case of a graph hang, we don't have
    to bother about other tracers.

    Thanks.



    \
     
     \ /
      Last update: 2009-03-21 21:49    [W:0.025 / U:30.228 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site