Messages in this thread | | | From | David Sharp <> | Date | Tue, 16 Nov 2010 15:27:58 -0800 | Subject | Re: Benchmarks of kernel tracing options (ftrace and ktrace) |
| |
On Wed, Oct 13, 2010 at 4:50 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > On Wed, 2010-10-13 at 16:19 -0700, David Sharp wrote: >> Google uses kernel tracing aggressively in the its data centers. We > > Thanks! > >> wrote our own kernel tracer, ktrace. However ftrace, perf and LTTng >> all have a better feature set than ktrace, so we are abandoning that >> code. > > Cool! > >> >> We see several implementations of tracing aimed at the mainline kernel >> and wanted a fair comparison of each of them to make sure they will >> not significantly impact performance. A tracing toolkit that is too >> expensive is not usable in our environment. >> > > [ snip for now (I'm traveling) ] > >> This first set of benchmark results compares ftrace to ktrace. The >> numbers below are the "on" result minus the "off" result for each >> configuration. >> >> ktrace: 200ns (tracepoint: kernel_getuid) >> ftrace: 224ns (tracepoint: timer:sys_getuid) >> ftrace: 587ns (tracepoint: syscalls:sys_enter_getuid) > > >> The last result shows that the syscall tracing is about twice as >> expensive as a normal tracepoint, which is interesting. > > Argh, the syscall tracing has a lot of overhead. There is only one > tracepoint that is hooked into the ptrace code, and will save all > registers before calling the functions. It enables tracing on all > syscalls and there's a table that decides whether or not to trace the > syscall. > > So I'm not surprised with the result that the syscall trace point is so > slow (note, perf uses the same infrastructure). > > -- Steve
Steve, Mike told me that you may have some patches to improve overhead of syscall tracing. I'd be interested in testing them. Can you send them my way?
Thanks, d# -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |