lkml.org 
[lkml]   [2004]   [Jul]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
Date
From
SubjectRe: [PATCH] IA64 audit support

First of all thanks for the detailed answer, I think this is an opportunity
to clarify a few things regarding LTT.

For the patch comparison to be useful we should only keep one instance of
the architecture-dependent portions of each patch as LTT has support for
6 architectures while audit only has 3.

Andrew Morton wrote:
> arch/i386/kernel/entry.S | 6
> arch/i386/kernel/ptrace.c | 10
...
> fs/namei.c | 15
> include/asm-i386/thread_info.h | 6
...
> include/linux/audit.h | 211 ++++++++
> include/linux/fs.h | 14
> include/linux/netlink.h | 1
> include/linux/sched.h | 3
> init/Kconfig | 20
> kernel/Makefile | 2
> kernel/audit.c | 825 ++++++++++++++++++++++++++++++++++
> kernel/auditsc.c | 922 +++++++++++++++++++++++++++++++++++++++
> kernel/fork.c | 10
> security/selinux/avc.c | 168 +++----
> security/selinux/include/avc.h | 7
> security/selinux/ss/services.c | 2
> 23 files changed, 2199 insertions(+), 144 deletions(-)

NOTE: The LTT patch you analyzed is fairly old. Here's a more recent
patch against 2.6.3 (attached) using the same criteria as above:
MAINTAINERS | 7
...
arch/i386/kernel/entry.S | 21
arch/i386/kernel/irq.c | 6
arch/i386/kernel/process.c | 9
arch/i386/kernel/sys_i386.c | 3
arch/i386/kernel/traps.c | 103 +
arch/i386/mm/fault.c | 21
...
fs/buffer.c | 3
fs/exec.c | 6
fs/ioctl.c | 6
fs/open.c | 10
fs/read_write.c | 36
fs/select.c | 9
...
include/asm-i386/ltt.h | 15
...
include/linux/ltt-core.h | 428 ++++++
include/linux/ltt-events.h | 424 ++++++
init/Kconfig | 32
ipc/msg.c | 2
ipc/sem.c | 3
ipc/shm.c | 3
kernel/Makefile | 1
kernel/exit.c | 5
kernel/fork.c | 3
kernel/itimer.c | 4
kernel/ltt-core.c | 2557 +++++++++++++++++++++++++++++++++++++++++
kernel/sched.c | 4
kernel/signal.c | 3
kernel/softirq.c | 11
kernel/time.c | 1
kernel/timer.c | 4
mm/filemap.c | 3
mm/memory.c | 4
mm/page_alloc.c | 4
mm/page_io.c | 2
net/core/dev.c | 5
net/socket.c | 9
71 files changed, 4442 insertions(+), 16 deletions(-)

> adds hooks all over the place.

You're right. However, it's worth looking at what's being added. So here's
an example from the scheduler:
@@ -1709,6 +1712,7 @@ switch_tasks:
++*switch_count;

prepare_arch_switch(rq, next);
+ TRACE_SCHEDCHANGE(prev, next);
prev = context_switch(rq, prev, next);
barrier();

If tracing is disabled, here's what this macro resolves to:
#define TRACE_SCHEDCHANGE(OUT, IN)
All the trace points added in fs/* ipc/* kernel/* mm/* net/* are the
same type of one-liners. Also, please note the number of lines changed
for each of the files in those directories. Typically, for each file,
there is one #include added and a trace statement such as the above.
As far as the architecture-dependent changes, they are of the same nature.
The only exception is the tracing of the system call entries and exits
which requires changes to entry.S and some C code that calls on the tracing
function that records the system call details, which in the case of LTT is
added to traps.c (hence the number of changes to that file as can be seen
in the above diffstat.)

Overall, the trace statements added could be found in any other Unix kernel.
They are not specific to the Linux kernel, but rather to the architecture of
the OS being analyzed by the end-user.

> The security code adds hooks everywhere too, but those deliver end-user
> functionality rather than being purely a developer support tool.
>
> Developer support tools are good, but are not as persuasive as end-user
> features. Because the audience is smaller, and developers know how to
> apply patches and rebuild stuff.

This is probably one of the biggest misconception about LTT amongst kernel
developers. So let me present this once more: LTT is _NOT_ for kernel
developers, it has never been developed with this crowd in mind. LTT is and
has _ALWAYS_ been intended for the end user.

The fact of the matter is that the events recorded by LTT are far too little
in detail to help in any sort of kernel debugging. Don't take my word for it:
I met Marcelo at OLS once and he recounted attempting to use LTT to track
things in the kernel and how he found it NOT to be good enough for what he
was doing. Ditto with Andrea.

How is this tool useful for the end user? Here's an excerpt from an e-mail I
sent to Andrea and a few other SuSE folks explaining this some time ago:
> What LTT is really good at, however, is to provide non-kernel gurus with an
> understanding of kernel dynamics. It is not reasonable to expect that every
> sysadmin will understand exactly how the kernel behaves and then rely on
> ktrace to isolate a problem (as I expect most kernel developers to be able
> to do). On the other hand, it is quite reasonable to expect sysadmins to be
> able to fire-up a tool which gives them a good idea of what's going on in a
> system. This may not help them find kernel bugs, but it will most certainly
> help them track down transient performance problems, and all other kernel-
> behavior-related bugs which are simply invisible to /proc, ps, and their
> friends.
>
> The same goes for developers tracking synchronization problems. gdb won't
> help, strace won't help, etc. because they rely on ptrace() which itself
> modifies application behavior ... same applies to printf() etc.etc.etc.
> There's an entire category of problems for which current user-space tools
> are not adapted for and kernel debugging tools (ktrace including) are
> simply overkill.

Generally speaking, there isn't a single tool out there that currently
exists that enables any end-user to understand the complex dynamic behavior
between the Linux kernel, his applications and the outside world. And as
you personally noted in the forward to Robert Love's book, the kernel is
only getting more complicated. Using the trace points added by the LTT
patch, the user-space utilities can provide a wealth of information to the
end-user that he cannot possibly collect in any other way. Here's some
example output from the user tools:

## Event type Time-stamp PID Description
Syscall entry 1,086,989,312,690,339 20 SYSCALL : setpgid; IP : 0x1000422C
Syscall exit 1,086,989,312,690,341 20
Trap entry 1,086,989,312,690,343 20 TRAP : Data Access; IP : 0x100097AC
Trap exit 1,086,989,312,690,400 20
Trap entry 1,086,989,312,690,406 20 TRAP : Data Access; IP : 0x1000A0DC
Trap exit 1,086,989,312,690,462 20
Trap entry 1,086,989,312,690,472 20 TRAP : Data Access; IP : 0x0FF21F70
Trap exit 1,086,989,312,690,533 20
Trap entry 1,086,989,312,690,540 20 TRAP : Data Access; IP : 0x1000A10C
Trap exit 1,086,989,312,690,595 20
Syscall entry 1,086,989,312,690,606 20 SYSCALL : getpgrp; IP : 0x10004654
Syscall exit 1,086,989,312,690,607 20
Syscall entry 1,086,989,312,690,610 20 SYSCALL : wait4; IP : 0x100099C0
Process 1,086,989,312,690,611 20 WAIT PID : -1
Sched change 1,086,989,312,690,615 29 IN : 29; OUT : 20; STATE : 1
Trap entry 1,086,989,312,690,624 29 TRAP : Data Access; IP : 0x10009584
Trap exit 1,086,989,312,690,628 29
Trap entry 1,086,989,312,690,631 29 TRAP : Data Access; IP : 0x100052EC
Trap exit 1,086,989,312,690,634 29
Syscall entry 1,086,989,312,690,644 29 SYSCALL : getpid; IP : 0x1000422C
Syscall exit 1,086,989,312,690,645 29
Syscall entry 1,086,989,312,690,647 29 SYSCALL : setpgid; IP : 0x1000422C
Syscall exit 1,086,989,312,690,648 29
Syscall entry 1,086,989,312,690,652 29 SYSCALL : ioctl; IP : 0x1000960C
File system 1,086,989,312,690,654 29 IOCTL : 2; COMMAND : 0x80047476

As you can see, the granularity of the details is not refined enough for any
sort of kernel debugging, yet it is clear that an end-user or an application
developer can benefit immensly from such information. Given the ever
increasing complexity of the kernel, the ever increasing number of applications
run on servers and workstations, and the ever increasing use of Linux in
time-sensitive applications such as embedded systems, it seems to me that this
type of capability is no less necessary then ptrace().

I'll conceed that LTT may be of some benefit for some driver developers in some
cases and that it may help consolidate the slew of tracing mechanisms
already included in the kernel as part of various drivers and subsystems, but
the fact of the matter is that it is of little use for kernel developers. If
a kernel developer needs tracing, he should be using ktrace.

> It's a balance between (ongoing maintenance cost multiplied by the number of
> impacted developers) versus (additional functionality multiplied by the
> number of users who benefit from it). To my mind, LTT (and kgdb and various
> other developer-support things) don't offer good ratios here.

Again, LTT is of marginal use to kernel developers, the benefits all go
to the end users' ability to understand what's going on in their system (see
above for examples.)

On the topic of maintenance cost, I fail to see how one-liners such as the
above can be of any burden to any kernel developer, they have remained
virtually unchanged for the past 5 years and any look throughout the LTT
archives or the kernel mailing list archive for LTT patches will readily
show this.

> If it could use kprobes hooks that'd be neat. kprobes is low-impact.

The issues about the spread of trace points across the source code are
exactly the same, you still need to mark the code-paths (and maintain
these markings for each version) regardless of the mechanism being used.
Not to mention that the whole idea of LTT is not to modify the kernel
behavior while, to the best of my understanding, kprobes relies on the
debug int ... Not to mention (#2) that once disabled, there is zero code
added to the kernel.

I submit to you that LTT's trace statements have not changed for the
past 5 years (since 2.2.x). They have not increased in number, nor changed
in their semantics. Here's a diffstat on 2.2.13 dated 18th of November 1999
(also attached):
Documentation/Configure.help | 37 +
Makefile | 4
arch/i386/config.in | 5
arch/i386/kernel/entry.S | 21
arch/i386/kernel/irq.c | 6
arch/i386/kernel/process.c | 6
arch/i386/kernel/sys_i386.c | 4
arch/i386/kernel/traps.c | 105 ++++
arch/i386/mm/fault.c | 10
drivers/Makefile | 10
drivers/trace/Makefile | 30 +
drivers/trace/tracer.c | 948 +++++++++++++++++++++++++++++++++++++++++++
drivers/trace/tracer.h | 99 ++++
fs/buffer.c | 4
fs/exec.c | 7
fs/ioctl.c | 6
fs/open.c | 10
fs/read_write.c | 34 +
fs/select.c | 10
include/linux/trace.h | 293 +++++++++++++
init/main.c | 10
ipc/msg.c | 3
ipc/sem.c | 3
ipc/shm.c | 3
kernel/Makefile | 4
kernel/exit.c | 6
kernel/fork.c | 6
kernel/itimer.c | 4
kernel/sched.c | 9
kernel/signal.c | 4
kernel/softirq.c | 6
kernel/trace.c | 189 ++++++++
mm/filemap.c | 4
mm/page_alloc.c | 7
mm/vmscan.c | 4
net/core/dev.c | 9
net/socket.c | 9
37 files changed, 1928 insertions(+), 1 deletion(-)
I believe that this is a very strong argument regarding the maintainability
of such statements, and it is my hope that, based on this evidence, the
kernel development community will recognize that the persistent fears
regarding the maintainbility of the LTT trace statements are unfounded.

If I overlooked something, please let me know.

Karim
--
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546[unhandled content-type:application/x-bzip2][unhandled content-type:application/x-bzip2]
\
 
 \ /
  Last update: 2005-03-22 14:04    [W:0.082 / U:0.460 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site