[lkml]   [2006]   [Oct]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [RFC] The New and Improved Logdev (now with kprobes!)

    On Thu, 5 Oct 2006, Mathieu Desnoyers wrote:

    > >
    > > Is this the static marks everyone is fighting over?
    > >
    > I consider that the previous discussion let to a general concensus where it has
    > been recognised that marking the code is generally acceptable and needed. As a
    > personal initiative following this discussion, I proposed a marker mechanism,
    > iterated for 2 weeks, and it is now at the 0.20 release. It went through about 3
    > complete rewrites during the process, but it seems that most objections has been
    > answered.

    Currently at 5932 unread messages, and I'm still reading that thread :-)

    A little birdy pointed me to which talks
    about this. Good to see it. I should have read the rest of the thread
    before posting, but this was originally about my logdev, and I got

    > This week, I ported LTTng to the marker mechanism by creating "probes", which
    > are the dynamically loadable modules that connects to the markers.


    > > > - Serialization mechanism (facilities) within probes (ltt-probes kernel
    > > > modules) dynamically connected to markers.
    > >
    > > Sorry, I don't really understand what the above is. Is it a loadable
    > > module that connects to the static markers? You might need to dumb this
    > > one down for me.
    > >
    > Ok,
    > A marker, in my implementation, is a statement placed in the code, i.e. :
    > MARK(kernel_sched_schedule, "%d %d %ld", prev->pid, next->pid, prev->state);
    > A probe is a dynamically loadable module which implements a callback, i.e. :
    > #define KERNEL_SCHED_SCHEDULE_FORMAT "%d %d %ld"
    > void probe_kernel_sched_schedule(const char *format, ...)
    > {
    > va_list ap;
    > /* Declare args */
    > int prev_pid, next_pid;
    > long state;
    > /* Assign args */
    > va_start(ap, format);
    > prev_pid = va_arg(ap, typeof(prev_pid));
    > next_pid = va_arg(ap, typeof(next_pid));
    > state = va_arg(ap, typeof(state));
    > /* Call tracer */
    > trace_process_schedchange(prev_pid, next_pid, state);
    > va_end(ap);
    > }
    > Which, in my case, takes the variable arguments of the marker call and gives
    > them to my inline tracing function.
    > trace_process_schedchange is the "serialization" mechanism which takes its input
    > (arguments) and writes them in a event record in the buffers, dealing with
    > reentrancy.

    OK, this makes a lot more sense. I still have a ton of questions, but they
    are probably answered in the 5932 messages I have yet to read/skim.

    > >
    > > > - Tracing control mechanism (ltt-tracer, ltt-control)
    > >
    > > Is this in kernel or tools?
    > >
    > I am absolutely not talking about the user space tools here, only kernel code.
    > This would be another discussion :)

    I didn't think you were, but I had to be sure.

    > In fact, ltt-control is the netlink interface that helps user space controlling
    > the tracer. It's not mandatory : the tracer can be controlled from within the
    > kernel too (useful for embedded systems).
    > > > - Buffer management mechanism (ltt-relay)
    > >
    > > So this uses the current relay system?
    > >
    > Yes, but I do my own synchronization. I implement my own reserve/commit and I
    > also export my own ioctl and poll to communicate with the user space buffer
    > reading daemon.

    Sounds like what logdev does too.

    > > The greatest resistance that I currently see with LTTng is the adding of
    > > static trace points. So if LTTng isn't fully crippled by working with
    > > dynamic addition of trace point (unmodifying the code), then try to get
    > > that in first. See below.
    > >
    > My first goal is to have the infrastructure in, without the instrumentation. And
    > yes, I want to connect this infrastructure with a nice dynamic instrumentation
    > tool like logdev or systemtap, as it will give fast usability of the
    > infrastructure to a user base.

    systemtap would obviously be prefered over my logdev (but I can still
    dream ;)

    > > > * Dynamic registration of new events/event record types
    > > >
    > > > LTTng supports such dynamic registration since the 0.5.x series.
    > >
    > > I feel really stupid! What do you define as an event, and how would
    > > one add a new one dynamically.
    > >
    > Please don't :) Defining a new event would be to say :
    > I want to create an event named "schedchange", which belongs to the "kernel"
    > subsystem. In its definition, I say that it will take two integers and a long,
    > respectively names "prev_pid", "next_pid" and "state".
    > We can think of various events for various subsystems, and even for modules. It
    > becomes interesting to have dynamically loadable event definitions which gets
    > loaded with kernel modules. The "description" of the events is saved in the
    > trace, in a special low traffic channel (small buffers with a separate file).

    But these events still need the marker in the source code right?

    > > >
    > > > * Probe placement
    > > >
    > > > What makes debugging information based probe placement unsuitable as the only
    > > > option for LTTng :
    > >
    > > First thing which is a key point: "only option" OK, while reading that
    > > nasty thread, I saw that LTTng can still function when certain features
    > > are not present. Basically, convert all posible static tracepoints into
    > > dynamic ones and make a code base for that. Have a patch to convert
    > > critical trace points that are not suitable for performance into static
    > > traces, and also add static traces that were not able to be done by
    > > dynamic ones. This way you have a functioning LTTng in the kernel (if the
    > > resistance falls by doing this), and still maintaing a patch for a "value
    > > added" to your customers. Perhaps call it "Turbo LTTng" ;-)
    > >
    > I won't try to convert all the existing "marker" based trace points into
    > dynamic ones, as I see no real use of it in the long run. However, I would
    > really like to see a kprobe based instrumentation being a little more nicely
    > integrated with LTTng (and it is not hard to do!).

    That should definitely be a step. If I'm understanding this (which I may
    not be), you can have a dynamic event added with also using dynamic
    trace points like kprobes.

    > > Basically, Mathieu, I want to help you get this into the kernel. I could
    > > be wrong, since I'm only a spectator, and not really involved on either
    > > side. But I have been reading LKML long enough to have an idea of what it
    > > takes.
    > >
    > > If you can modulize LTTng further down. Add non intrusive parts to the
    > > kernel. If you can make a LTTng functional (but "crippled" due to the
    > > limitations you are saying) and have it doing what the ney-sayers want,
    > > you will have a better time getting it accepted. Once accepted, it will
    > > be a lot easier to add controversial things than it is to add it before
    > > any of it is accepted.
    > >
    > Yes, that's why I am splitting my projects in parts "markers, tracer, control,
    > facilities, ..." and plan to keep the most intrusives as an external patchset
    > for the moment. Anyway, the marker-probe mechanism lets me put all the
    > serialization code inside probes external to the kernel, which can be connected
    > either with the "marker" mechanism or with kprobes, it doesn't matter.
    > Thanks for your hints,

    No prob, I should read the rest of the thread, and try to catch up more,
    before posting more comments.


    -- Steve

    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2006-10-05 23:33    [W:0.031 / U:11.456 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site