lkml.org 
[lkml]   [2009]   [Apr]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: BUG: Function graph tracer hang


On Wed, 29 Apr 2009, Ingo Molnar wrote:

>
> * Steven Rostedt <rostedt@goodmis.org> wrote:
>
> >
> > On Tue, 28 Apr 2009, Frederic Weisbecker wrote:
> >
> > >
> > > Note that the branch profiler does that:
> > >
> > > ______f.miss_hit[______r]++;
> > >
> > > Which is a read + write on the cacheline.
> > > If each "if" are profiled in the timer interrupt, we can
> > > have the cachelines doing a ping-pong of dirtifying since the above
> > > variable is shared.
> > >
> > > Then the timer interrupt becomes slower. The function graph tracer itself makes
> > > it slower.
> > > Moreover it is traced itself. So not only the "if" in code are traced, but also
> > > each "if" processed by the function graph tracer on function calls and returns.
> > >
> > > Which means a fair amount of cacheline dirtifying.
> > >
> > > Then if the timer interrupt is slowed, and we have a lot of them (1000 Hz),
> > > the system spends all of its time inside it.
> > >
> > > At least we need the branch tracing to be done per cpu, I guess.
> >
> > This can be done by basically reimplementing what percpu does.
> > This is because the data is saved off in its own section at every
> > if statement. We could copy that section per cpu and add code to
> > the incrementors to add only to their own CPU buffers.
>
> Why not just make these variables regular percpu constructs?

Because it is a special section. The variables are embedded into the "if"
statement. We would need to add it to both the percpu section as well as
its own section.
One section for the per cpu data, the other to print out the data. We use
the section to gather all the if statements. Hmm, maybe we can simply make
the linker script put the if section within the percpu section?

>
> > This is low in my priority of things to do, but I'll at least add
> > it to my "to do" list.
>
> I think we should mark the branch tracer as CONFIG_BROKEN - there's
> been too many problems with it. Thoughts?

You mean the branch profiler? I've been using it without any issues. The
issues usually come when we have it combined with other tracers on
large(?) SMP boxes. I don't hit the cacheline bouncing hang on my 4 cores.

-- Steve



\
 
 \ /
  Last update: 2009-04-29 14:27    [from the cache]
©2003-2011 Jasper Spaans