Messages in this thread | | | Date | Mon, 02 Nov 2009 10:02:23 -0500 | From | Masami Hiramatsu <> | Subject | Re: [PATCH 0/3][RFC] tracing/kprobes: prevent jprobes from crashing function graph tracer |
| |
Frederic Weisbecker wrote: > On Thu, Oct 29, 2009 at 06:02:20PM -0400, Masami Hiramatsu wrote: >> Steven Rostedt wrote: >>> Lately I've been testing with an allyesconfig. When I ran the function graph >>> tracer, it immediately crashed the kernel. Thanks to the new frame pointer >>> test in function graph, it reported directly what the issue was and then >>> panicked the kernel to prevent any unexpected damage from happening. >>> >>> It pointed the error to be with jtcp_rcv_established. Which is a jprobe >>> function added to tcp_rcv_established at bootup when CONFIG_NET_TCPPROBE >>> is enabled. >>> >>> Jprobes and the function graph tracer use the same mechanism to trace >>> the exit of a function. Unfortunately, only one can be done at a time. >>> The function graph tracer replaces the return address with its own handler, >>> but so does jprobes. The two are not compatible. >> >> AFAIK, Jprobe doesn't trace the exit of a function. I assume that >> jprobe's user handler causes the problem, since the handler never >> returns normal way. >> Instead of that, it just calls jprobe_return() which causes >> int3 to be trapped by kprobe's break handler. And the break handler >> fixup regs->ip to back to traced function. >> >> Actually, this will cause a problem with function graph tracer. >> The f-g-tracer push the return address into the special stack and replaces >> it with fixup function (This is similar (not same) mechanism of kretprobe.) >> And then the traced function returns, it returns to the fixup function and >> it pops the return address up and back to the real caller. >> >> So, if the f-g-tracer traces jprobe user handler, the pop operation >> will be skipped because the the handler never returns. > > > I'm not sure I've well understood how is performed the call to the jprobe > handler. > But if I understand well we have: > > func() { > int3() { > jprobe_handler() { > (-) > set ip after iret to user_handler() > } > } > user_handler() { > jprobe_return() { > (+) > int3() { > set ip after iret to func+...() > } > | > | > | > <-------------- > (execute the rest of func()) > } > > If we replace (-) with pause_graph_tracing() and (+) with > unpause_graph_tracing(), this should do the trick...I hope.
I'm not so sure about pause_graph_tracing(), however, it seems that int3() and jprobe_handler() already pushed on the stack of the func graph tracer at (-). If it's true, where are those entries popped up?
Thank you,
-- Masami Hiramatsu
Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division
e-mail: mhiramat@redhat.com
| |