Messages in this thread | | | Subject | [ANNOUNCE] Updated Logdev for internal Linux kernel debugging | From | Steven Rostedt <> | Date | Mon, 05 Nov 2007 09:32:50 -0500 |
| |
Finally, I was able to spend some time to clean up logdev for the latest kernels (2.6.23.1, 2.6.24-rc1"x86 merged" and 2.6.23.1-rt5 "Real-Time").
http://rostedt.homelinux.com/logdev/
What is logdev?
It is an internal ring buffered log trace system, to help debug those really hard to debug areas, like interrupts and the scheduler. It uses an internal memory ring buffer to load the information to and it will dump it out on a system panic, oops, nmi lockup, and most other crashes. You can also trigger the dump with a "sysrq-g" or read it straight from userland utilities.
What makes logdev special
"Everyone has written their own memory logging device" - tglx
logdev is special because I just happen not to delete the internal memory device when I was done with it and I've actively maintained it since early 2.2 kernels.
This means that I tried hard to make it as easy to port as possible. So it is very non intrusive in the kernel. When debugging a kernel, the first thing I usually do when the bugs are not obvious is to port logdev to that kernel.
Once logdev is there, there are several options to make it easy to trace.
One add the logdev.h header
#include <linux/logdev.h>
then simply use the macros to recode what you want in printk like statements.
lfcnprint("hello world");
can produce this:
[ 15.177033] cpu:1 (swapper:1) do_basic_setup:742 hello world
It shows you the timestamp, the CPU that it was called on, the process "comm" and pid, as well as the function and line number where the lfcnprint was made, and then the printk like string.
This debugging tool has become invaluable for debugging for me (and others http://www.gatago.org/linux/kernel/53143673.html)
I don't care if you use it or not, but since I find it so helpful, I like to share it with anyone else that might find it useful too.
For more details on use look here: http://rostedt.homelinux.com/logdev/README
Why not use relay (aka. relayfs)?
After posting this a while ago, Ingo Molnar suggested that I look into a similar utility called relayfs (now called relay). I started some correspondence with Tom Zanussi about using relay as my back end for logdev.
Unfortunately, relay is focused very much so for fast interaction with userland. It is split up in pages that are mapped special so that userland can map these pages and do the consumer work too. It is quite complex to handle all the interactions that are needed for working with and efficiently with userland.
relay is for things like LTT which is a complex tool to help developers and users alike analyze their systems.
logdev is for debugging. It should never be turned on for production systems.
One advantage that logdev has over relay is that, since it is specific for debugging, it isn't such a high priority to be efficient. So it can interleave its per cpu buffers with a single atomic counter so that the output is guaranteed to be in the order of occurrences. logdev never trusts timestamps for this purpose. But its for debugging, so who cares about a little cache line bouncing ;-)
Also, Tom Zanussi was kind enough to spend some time trying to make relay a back end for logdev (still there). But because logdev is so kernel centric and relay is very much user centric, the two didn't seem to fit well at all.
One thing that logdev still does nicely is that it dumps its buffers out to console (printk) whenever there's a bug or crash. This happens to be the most useful part that I find, and the part that I use most often. When I'm developing a feature in the kernel scheduler or something, and I'm crashing or locking up, adding logdev to it shows nicely what is happening, and because its a ring buffer, you see what has happened most recently, which usually includes the area that has the bug.
I even used logdev to prove to tglx that there was a nasty race condition in his early versions of hrtimers.
Why not use Mathieu's Markers?
Heh, well actually, I want to ;-)
I'm just waiting to see the final version of them when they are in 2.6.24. After that, I'll add hooks into logdev to use them. Currently, logdev has markers that are similar, but they were only created as a proof of concept for Mathieu to try out some other ideas. The logdev markers are obviously not in mainline.
Do you want logdev to be in the Linus tree?
No. (really, I don't). The Linus tree is for distributions to take and put into production products. As I said, logdev is a nice tool for kernel hackers, but not much else. For users that want the features of logdev, relay is a much better way to go. I'm posting this so other kernel hackers can play.
That said...
Do you want logdev to be in Andrew Mortons -mm tree?
Yes! Well, it's really up to Andrew. But I remember him once saying, that if there was a tool to help debugging, and wasn't meant to be in Linus's tree, but it was non intrusive and well maintained, he would consider holding it in the -mm tree.
Well, logdev is for hackers and debuggers, and the -mm tree seems to fit that category, not to mention that logdev is easily ported form one kernel to the next (I've been doing it since 2.2). I think this might be a good place for it to live.
But this is really up to Andrew to decide.
Well, that's it. Comments, questions, flames, all welcomed.
-- Steve
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |