Messages in this thread | | | Date | Mon, 26 Nov 2018 18:21:12 +0900 | From | Masami Hiramatsu <> | Subject | Re: [RFC][PATCH 00/14] function_graph: Rewrite to allow multiple users |
| |
Hi Steve,
On Wed, 21 Nov 2018 20:27:08 -0500 Steven Rostedt <rostedt@goodmis.org> wrote:
> > I talked with many of you at Plumbers about rewriting the function graph > tracer. Well, this is it. I was originally going to produce just a > proof of concept, but when I found that I had to fix a design flaw > and that covered all the arch code anyway, I decided to do more of a > RFC patch set.
Thank you for starting this work! This might be a good way to simplify treating of shadow stacks in kretprobe and function-graph-tracers. (And I hope this can help me to remove some kind of complexity in kretprobes)
> > I probably should add more comments to the code, and update the > function graph design documentation, but I wanted to get this out > before the US Turkey day for your enjoyment while you try to let your > pants buckle again.
:)
Let me try to review and port kretprobe on it. On the way, I will find issues if there are.
> > Why the rewrite? > > Well the fuction graph tracer is arguably the strongest of the tracers. > It shows both the entrance and exit of a function, can give the timings > of a function, and shows the execution of the code quite nicely. > > But it has one major flaw. > > It can't let more than one user access it at a time. The function > tracer has had that feature for years now, but due to the design of > the function graph tracer it was difficult to implement. Why? > > Because you must maintain the state of a three-tuple. > > Task, Function, Callback > > The state is determined at by the entryfunc and must be passed to the > retfunc when the function being traced returns. But this is not an > easy task, as that state can be different for each task, each function > and each callback. > > What's the solution? I use the shadow stack that is already being > used to store the function return addresses. > > A big thanks to Masami Hiramatsu for suggesting this idea! > > For now, I only allow an 16 users of the function graph tracer at a time. > That should be more than enough. I create an array of 16 fgraph_ops > pointers. When a user registers their fgraph_ops to the function graph > tracer, it is assigned an index into that array, which will hold a pointer > to the fgraph_ops being registered. > > On entry of the function, the array is iterated and each entryfunc of > the fgraph_ops in the array is called. If the entryfunc returns non-zero, > then the index of that fgraph_ops is pushed on the shadow stack (along > with the index to the "ret_stack entry" structure, for fast access > to it). If the entryfunc returns zero, then it is ignored. If at least > one function returned non-zero then the return of the traced function > will also be traced. > > On the return of the function, the shadow stack is examined and all > the indexes that were pushed on the stack is read, and each fgraph_ops > retfunc is called in the reverse order. > > When a fgraph_ops is unregistered, its index in the array is set to point > to a "stub" fgraph_ops that holds stub functions that just return > "0" for the entryfunc and does nothing for the retfunc. This is because > the retfunc may be called literally days after the entryfunc is called > and we want to be able to free the fgraph_ops that is unregistered. > > Note, if another fgraph_ops is registered in the same location, its > retfunc may be called that was set by a previous fgraph_ops. This > is not a regression because that's what can happen today if you unregister > a callback from the current function_graph tracer and register another > one. If this is an issue, there are ways to solve it.
Yeah, I need the solution, maybe an API to get correct return address? :)
By the way, are there any way to hold a private data on each ret_stack entry? Since kretprobe supports "entry data" passed from entry_handler to return handler, we have to store the data or data-instance on the ret_stack.
This feature is used by systemtap to save the function entry data, like function parameters etc., so that return handler analyzes the parameters with return value.
Thank you,
-- Masami Hiramatsu <mhiramat@kernel.org>
| |