lkml.org 
[lkml]   [2015]   [Jul]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH v9 00/22] tracing: 'hist' triggers
From
Date
Hi Brendan,

On Wed, 2015-07-22 at 11:29 -0700, Brendan Gregg wrote:
> G'Day Tom,
>
> On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi
> <tom.zanussi@linux.intel.com> wrote:
> >
> > This is v9 of the 'hist triggers' patchset.
> >
> [...]
>
> I've browsed the functionality (sorry, catching up), and it looks like
> this will solve a number of common problems. But it seems
> tantalizingly close to solving a few more. These may already be on
> your future todo list.
>
> A) CPU stack profiling
>
> Kernel stacktrace as a key will be hugely useful; is there a way to
> enable this for a sampling profile? (eg, what perf record -F 99 does).
> I take CPU profiles daily, and would prefer to aggregate stacks
> in-kernel. Also, would like user stacktrace as a key (even if it's
> just the hex).
>

This wasn't on my todo list but I can see how it would be useful. On
the list now. ;-)

> B) Key buckets
>
> Eg, imagine:
>
> echo 'hist:keys=common_pid.execname,count.log2:val=count' >
> /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
>
> to get a log2 bucketized histogram of syscall read request size. Same
> for any value where using the value as a key gets too verbose, and you
> just want a rough look at the distribution. (Would make it more
> readable if it could also be sorted by the log2 value.)
>

This I actually had an early implementation of, which I plan on
reviving...

> C) Latency as a bucket key
>
> With kprobes, we could then have a log2 histogram of any function call
> latency, collected efficiently. (There's already the function timers
> in ftrace, which I'm using from function_graph with filters sets to
> only match the target function.)
>

My original thought for doing this kind of thing was to generalize the
concept of a 'difference'. I used it in v1 as a way to calculate the
difference between requested and allocated sizes for memory allocations,
which was kind of pointless, though convenient. The real value would of
course be in applying it to inter-event values rather than intra-event.
In this case it would be an inter-event difference between timestamps.

And in my previous patchset, I had a 'function_hist' tracer, similar to
and in fact based on the same code as function_graph, that would simply
aggregate hitcounts for every function call in the kernel, which indeed
was pretty efficient. So the pieces are or have been there to do
something like this, just a matter of putting them together.

> ... Those are the other common use cases, that the hist functionality
> seemed suited for. Beyond that gets more custom, and we can use eBPF.
>

Exactly.

Tom

> Brendan




\
 
 \ /
  Last update: 2015-07-23 04:21    [W:2.128 / U:1.180 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site