lkml.org 
[lkml]   [2009]   [Jul]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH 1/2] ftrace: Add duration filtering to function graph tracer

On Mon, 6 Jul 2009, Tim Bird wrote:

> Steven Rostedt wrote:
> > On Mon, 6 Jul 2009, Tim Bird wrote:
> >
> >> Add duration filtering to the function graph tracer.
> >>
> >> The duration filter value is set in the 'tracing_thresh'
> >> pseudo-file. Values are in microseconds (as is customary
> >> for that file).
> >>
> >> This adds ring_buffer_peek_previous(), used to help
> >> remove the function entry event from the trace log,
> >> where this is possible.
> >>
> >> To use:
> >> $ cd <debugfs tracing directory>
> >> $ echo 100 >tracing_thresh
> >> $ echo function_graph >current_tracer
> >> $ cat trace
> >
> > I see what you are trying to do, but this can be really dangerous.
> > Remember, the ring buffer is now lockless. This could probably cause some
> > problems with various races.
>
> That's something I'm worried about.
>
> Note that this patch only uses ring_buffer_peek_previous (which doesn't
> alter anything in the log), and ring_buffer_event_discard(), which should
> be atomic on "blotting out" the entry. Obviously, a change of page
> contents between the two would make things interesting, but since
> this is in the committed area of a page, that seems really unlikely.
>
> However, the truly dangerous stuff is in updating the commit pointer.
> (ring_buffer_rewind_tail in patch 2/2).
>
> As near as I can tell, that should be safe when a reader is not
> going at the same time as a writer. In my use cases, I don't let
> readers and writers go at the same time (that is, the trace is always
> stopped when I'm dumping it.) I'm not sure if this is an acceptable
> condition to put on use of this feature or not, but it it was found
> to guarantee safeness, it could be enforced via the user interface.

There's no guarantee that readers will not be going at the same time
as writers. I tried hard to allow the ring buffer to accept a reader on
another CPU while writers were happening. A writer may even preempt a
reader on the same CPU.

>
> > If you want a duration field in the function graph tracer, perhaps only do
> > the recording on the exit side. That may be tricky since you would also
> > need to keep the stack order as well.
>
> This might work.
>
> For a single process, I have calling order in ret_stack. I also have calltime,
> which should be granular enough to disentangle the call starts for functions
> from different processes. It might need a post-trace reprocessor to fix up
> the results, though.
>
> > Perhaps implement an auxiliary ring buffer?
> This is a possibility. Are you thinking of something like double-buffering
> the events?
>
> Another thing I thought of was to not commit the entry event until function
> exit. I'm not sure the ring buffer supports having an entry outstanding for
> long periods of time, though. This would, I believe, hold readers at the entries for
> the last 'completed' functions, which might solve reader/writer races.

Heh, I doubt that would be acceptable. The problem is that between reserve
and commit, we disable preemption. Thus every function (even the scheduler
itself) would have preemption disabled ;-)

-- Steve


>
> I should add, that although this stuff looks dangerous, it's working pretty
> well for me here. As a debug tool, I could tolerate the occasional hang.
> I'm not seeing any so far, but to be honest I haven't really pounded hard
> on it yet.

>
> =============================
> Tim Bird
> Architecture Group Chair, CE Linux Forum
> Senior Staff Engineer, Sony Corporation of America
> =============================
>
>


\
 
 \ /
  Last update: 2009-07-07 04:19    [W:0.346 / U:0.352 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site