Messages in this thread Patch in this message | | | Date | Thu, 01 Jul 2004 23:21:17 -0400 | From | Karim Yaghmour <> | Subject | Re: [PATCH] IA64 audit support |
| |
First of all thanks for the detailed answer, I think this is an opportunity to clarify a few things regarding LTT.
For the patch comparison to be useful we should only keep one instance of the architecture-dependent portions of each patch as LTT has support for 6 architectures while audit only has 3.
Andrew Morton wrote: > arch/i386/kernel/entry.S | 6 > arch/i386/kernel/ptrace.c | 10 ... > fs/namei.c | 15 > include/asm-i386/thread_info.h | 6 ... > include/linux/audit.h | 211 ++++++++ > include/linux/fs.h | 14 > include/linux/netlink.h | 1 > include/linux/sched.h | 3 > init/Kconfig | 20 > kernel/Makefile | 2 > kernel/audit.c | 825 ++++++++++++++++++++++++++++++++++ > kernel/auditsc.c | 922 +++++++++++++++++++++++++++++++++++++++ > kernel/fork.c | 10 > security/selinux/avc.c | 168 +++---- > security/selinux/include/avc.h | 7 > security/selinux/ss/services.c | 2 > 23 files changed, 2199 insertions(+), 144 deletions(-)
NOTE: The LTT patch you analyzed is fairly old. Here's a more recent patch against 2.6.3 (attached) using the same criteria as above: MAINTAINERS | 7 ... arch/i386/kernel/entry.S | 21 arch/i386/kernel/irq.c | 6 arch/i386/kernel/process.c | 9 arch/i386/kernel/sys_i386.c | 3 arch/i386/kernel/traps.c | 103 + arch/i386/mm/fault.c | 21 ... fs/buffer.c | 3 fs/exec.c | 6 fs/ioctl.c | 6 fs/open.c | 10 fs/read_write.c | 36 fs/select.c | 9 ... include/asm-i386/ltt.h | 15 ... include/linux/ltt-core.h | 428 ++++++ include/linux/ltt-events.h | 424 ++++++ init/Kconfig | 32 ipc/msg.c | 2 ipc/sem.c | 3 ipc/shm.c | 3 kernel/Makefile | 1 kernel/exit.c | 5 kernel/fork.c | 3 kernel/itimer.c | 4 kernel/ltt-core.c | 2557 +++++++++++++++++++++++++++++++++++++++++ kernel/sched.c | 4 kernel/signal.c | 3 kernel/softirq.c | 11 kernel/time.c | 1 kernel/timer.c | 4 mm/filemap.c | 3 mm/memory.c | 4 mm/page_alloc.c | 4 mm/page_io.c | 2 net/core/dev.c | 5 net/socket.c | 9 71 files changed, 4442 insertions(+), 16 deletions(-)
> adds hooks all over the place.
You're right. However, it's worth looking at what's being added. So here's an example from the scheduler: @@ -1709,6 +1712,7 @@ switch_tasks: ++*switch_count;
prepare_arch_switch(rq, next); + TRACE_SCHEDCHANGE(prev, next); prev = context_switch(rq, prev, next); barrier();
If tracing is disabled, here's what this macro resolves to: #define TRACE_SCHEDCHANGE(OUT, IN) All the trace points added in fs/* ipc/* kernel/* mm/* net/* are the same type of one-liners. Also, please note the number of lines changed for each of the files in those directories. Typically, for each file, there is one #include added and a trace statement such as the above. As far as the architecture-dependent changes, they are of the same nature. The only exception is the tracing of the system call entries and exits which requires changes to entry.S and some C code that calls on the tracing function that records the system call details, which in the case of LTT is added to traps.c (hence the number of changes to that file as can be seen in the above diffstat.)
Overall, the trace statements added could be found in any other Unix kernel. They are not specific to the Linux kernel, but rather to the architecture of the OS being analyzed by the end-user.
> The security code adds hooks everywhere too, but those deliver end-user > functionality rather than being purely a developer support tool. > > Developer support tools are good, but are not as persuasive as end-user > features. Because the audience is smaller, and developers know how to > apply patches and rebuild stuff.
This is probably one of the biggest misconception about LTT amongst kernel developers. So let me present this once more: LTT is _NOT_ for kernel developers, it has never been developed with this crowd in mind. LTT is and has _ALWAYS_ been intended for the end user.
The fact of the matter is that the events recorded by LTT are far too little in detail to help in any sort of kernel debugging. Don't take my word for it: I met Marcelo at OLS once and he recounted attempting to use LTT to track things in the kernel and how he found it NOT to be good enough for what he was doing. Ditto with Andrea.
How is this tool useful for the end user? Here's an excerpt from an e-mail I sent to Andrea and a few other SuSE folks explaining this some time ago: > What LTT is really good at, however, is to provide non-kernel gurus with an > understanding of kernel dynamics. It is not reasonable to expect that every > sysadmin will understand exactly how the kernel behaves and then rely on > ktrace to isolate a problem (as I expect most kernel developers to be able > to do). On the other hand, it is quite reasonable to expect sysadmins to be > able to fire-up a tool which gives them a good idea of what's going on in a > system. This may not help them find kernel bugs, but it will most certainly > help them track down transient performance problems, and all other kernel- > behavior-related bugs which are simply invisible to /proc, ps, and their > friends. > > The same goes for developers tracking synchronization problems. gdb won't > help, strace won't help, etc. because they rely on ptrace() which itself > modifies application behavior ... same applies to printf() etc.etc.etc. > There's an entire category of problems for which current user-space tools > are not adapted for and kernel debugging tools (ktrace including) are > simply overkill.
Generally speaking, there isn't a single tool out there that currently exists that enables any end-user to understand the complex dynamic behavior between the Linux kernel, his applications and the outside world. And as you personally noted in the forward to Robert Love's book, the kernel is only getting more complicated. Using the trace points added by the LTT patch, the user-space utilities can provide a wealth of information to the end-user that he cannot possibly collect in any other way. Here's some example output from the user tools:
## Event type Time-stamp PID Description Syscall entry 1,086,989,312,690,339 20 SYSCALL : setpgid; IP : 0x1000422C Syscall exit 1,086,989,312,690,341 20 Trap entry 1,086,989,312,690,343 20 TRAP : Data Access; IP : 0x100097AC Trap exit 1,086,989,312,690,400 20 Trap entry 1,086,989,312,690,406 20 TRAP : Data Access; IP : 0x1000A0DC Trap exit 1,086,989,312,690,462 20 Trap entry 1,086,989,312,690,472 20 TRAP : Data Access; IP : 0x0FF21F70 Trap exit 1,086,989,312,690,533 20 Trap entry 1,086,989,312,690,540 20 TRAP : Data Access; IP : 0x1000A10C Trap exit 1,086,989,312,690,595 20 Syscall entry 1,086,989,312,690,606 20 SYSCALL : getpgrp; IP : 0x10004654 Syscall exit 1,086,989,312,690,607 20 Syscall entry 1,086,989,312,690,610 20 SYSCALL : wait4; IP : 0x100099C0 Process 1,086,989,312,690,611 20 WAIT PID : -1 Sched change 1,086,989,312,690,615 29 IN : 29; OUT : 20; STATE : 1 Trap entry 1,086,989,312,690,624 29 TRAP : Data Access; IP : 0x10009584 Trap exit 1,086,989,312,690,628 29 Trap entry 1,086,989,312,690,631 29 TRAP : Data Access; IP : 0x100052EC Trap exit 1,086,989,312,690,634 29 Syscall entry 1,086,989,312,690,644 29 SYSCALL : getpid; IP : 0x1000422C Syscall exit 1,086,989,312,690,645 29 Syscall entry 1,086,989,312,690,647 29 SYSCALL : setpgid; IP : 0x1000422C Syscall exit 1,086,989,312,690,648 29 Syscall entry 1,086,989,312,690,652 29 SYSCALL : ioctl; IP : 0x1000960C File system 1,086,989,312,690,654 29 IOCTL : 2; COMMAND : 0x80047476
As you can see, the granularity of the details is not refined enough for any sort of kernel debugging, yet it is clear that an end-user or an application developer can benefit immensly from such information. Given the ever increasing complexity of the kernel, the ever increasing number of applications run on servers and workstations, and the ever increasing use of Linux in time-sensitive applications such as embedded systems, it seems to me that this type of capability is no less necessary then ptrace().
I'll conceed that LTT may be of some benefit for some driver developers in some cases and that it may help consolidate the slew of tracing mechanisms already included in the kernel as part of various drivers and subsystems, but the fact of the matter is that it is of little use for kernel developers. If a kernel developer needs tracing, he should be using ktrace.
> It's a balance between (ongoing maintenance cost multiplied by the number of > impacted developers) versus (additional functionality multiplied by the > number of users who benefit from it). To my mind, LTT (and kgdb and various > other developer-support things) don't offer good ratios here.
Again, LTT is of marginal use to kernel developers, the benefits all go to the end users' ability to understand what's going on in their system (see above for examples.)
On the topic of maintenance cost, I fail to see how one-liners such as the above can be of any burden to any kernel developer, they have remained virtually unchanged for the past 5 years and any look throughout the LTT archives or the kernel mailing list archive for LTT patches will readily show this.
> If it could use kprobes hooks that'd be neat. kprobes is low-impact.
The issues about the spread of trace points across the source code are exactly the same, you still need to mark the code-paths (and maintain these markings for each version) regardless of the mechanism being used. Not to mention that the whole idea of LTT is not to modify the kernel behavior while, to the best of my understanding, kprobes relies on the debug int ... Not to mention (#2) that once disabled, there is zero code added to the kernel.
I submit to you that LTT's trace statements have not changed for the past 5 years (since 2.2.x). They have not increased in number, nor changed in their semantics. Here's a diffstat on 2.2.13 dated 18th of November 1999 (also attached): Documentation/Configure.help | 37 + Makefile | 4 arch/i386/config.in | 5 arch/i386/kernel/entry.S | 21 arch/i386/kernel/irq.c | 6 arch/i386/kernel/process.c | 6 arch/i386/kernel/sys_i386.c | 4 arch/i386/kernel/traps.c | 105 ++++ arch/i386/mm/fault.c | 10 drivers/Makefile | 10 drivers/trace/Makefile | 30 + drivers/trace/tracer.c | 948 +++++++++++++++++++++++++++++++++++++++++++ drivers/trace/tracer.h | 99 ++++ fs/buffer.c | 4 fs/exec.c | 7 fs/ioctl.c | 6 fs/open.c | 10 fs/read_write.c | 34 + fs/select.c | 10 include/linux/trace.h | 293 +++++++++++++ init/main.c | 10 ipc/msg.c | 3 ipc/sem.c | 3 ipc/shm.c | 3 kernel/Makefile | 4 kernel/exit.c | 6 kernel/fork.c | 6 kernel/itimer.c | 4 kernel/sched.c | 9 kernel/signal.c | 4 kernel/softirq.c | 6 kernel/trace.c | 189 ++++++++ mm/filemap.c | 4 mm/page_alloc.c | 7 mm/vmscan.c | 4 net/core/dev.c | 9 net/socket.c | 9 37 files changed, 1928 insertions(+), 1 deletion(-) I believe that this is a very strong argument regarding the maintainability of such statements, and it is my hope that, based on this evidence, the kernel development community will recognize that the persistent fears regarding the maintainbility of the LTT trace statements are unfounded.
If I overlooked something, please let me know.
Karim -- Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || karim@opersys.com || 1-866-677-4546[unhandled content-type:application/x-bzip2][unhandled content-type:application/x-bzip2]
| |