[lkml]   [2009]   [Jan]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectALLANC: Proposal to add progress records to process accounting

    I wanted to float an idea to get the community view before trying to
    implement something in the kernel for possible submission. I've structured
    this post into short description, long description and initial technical
    analysis so you can just stop reading when you get bored. Any suggestions
    on the value of this proposal, or improvements to the analysis, would be
    greatly appreciated. I'd say "be gentle" since this is my first foray onto
    LKML, but I'm willing to wear whatever scorn is heaped upon me :-).

    Short description:

    Change process accounting to output 'progress' records for long-lived
    processes (in addition to current records on process termination).

    Longer description:

    We (more specifically, the customers of our product) have been using
    process accounting for chargeback (departments are charged based on their
    usage) and performance (capacity planning) purposes. A heavily used
    application, IBM Websphere Application Server [bless their little hearts
    :-( ], recently changed its behaviour from "JVM processes exit regularly to
    be replaced by others" to "JVM processes hang around for weeks without
    exiting". So now the process accounting records aren't being written
    frequently. I realize this is our problem, not yours, but bear with me, the
    solution may be beneficial for others.

    We currently suggest the workaround that customers periodically restart WAS
    itself so that records are written more frequently. One customer has even
    taken to monitoring /proc/PID/stat but that's a bit of a kludge.

    This proposal is a suggestion to add a new type of record to process
    accounting that gives periodic updates of a process while it's still
    active, rather than waiting for process exit (the exit record is still
    written). It appears to me that there must be many long-lived processes
    that could benefit from this change, inasmuch as they could be monitored
    during their lifetime rather than only at exit.

    Technical analysis:

    The BSD process accounting within Linux writes type-3 records to the
    process accounting file when a process exits (this is done via a call to
    kernel/acct.c/acct_process() in kernel/exit.c/do_exit()).

    The proposal is to add another type (type-6 since 1-5 are taken, I believe)
    which is a incremental view of the process statistics, i.e, what's been
    added since the last incremental view (or process startup if this is the
    first increment). These records are exactly the same format as type-3 and
    will be written for any process that hasn't written a record in a while
    (for our purposes, 24 hours is fine but it may be better to make this

    There are two types of fields in the record, snapshot (such as ac_uid,
    ac_gid, ac_tty, ac_btime, ac_mem) and cumulative (such as ac_stime,
    ac_etime). The cumulative fields would only have the difference between the
    last type-6 and current values, so that you're not reliant on working out
    the values by referring to the previous type-6.

    So, let's say we have a process that runs for an 45 minutes and we have our
    configurable time set to 20 minutes. Further say process accounting only
    output PID and user/system/elapsed time (to make this next bit easier to

    At time t, process 42 starts.
    At time t+20min, process 42 has used 4 seconds system time, 12 seconds user
    time and outputs:
    rectype=6,pid=42, etime=1200, stime=4, utime=12
    At time t+40min, process 42 has used 19 seconds system time (since start),
    38 seconds user time (since start) and outputs:
    rectype=6,pid=42, etime=1200, stime=15, utime=26
    At time t+45min, process 42 exits, having used 20 seconds system time
    (since start), 40 seconds user time (since start) and outputs:
    rectype=6,pid=42, etime=300, stime=1, utime=2
    rectype=3,pid=42, etime=2700, stime=20, utime=40

    So, as you can see, the type-3 is still output on process exit, not
    breaking any programs (such as dump_acct) that expect them. However,
    programs which want a more timely view of the process activity can look at
    the type-6 records.

    How to implement it, that's the tricky bit. My first thought is to do
    checks on process context switch. Any process that becomes inactive and
    hasn't written a type-6 for too long will immediately write its type-6
    before the next process starts running. We'd need to ensure we don't add
    too much overhead to the context switch (that's putting it mildly). I
    assume this would be somewhere in kernel/sched.c/context_switch() or
    kernel/sched.c/prepare_task_switch() although I haven't yet looked in great

    The code that currently outputs the type-3 (kernel/acct.c/acct_process())
    would also output a type-6 first so the type-6 information is up to date
    with the type-3.

    From what I can see, the process accounting information can be reached from
    the process descriptor stored with the process kernel stack
    (current->signal->pacct) so there may have to be some more space allocated
    there for incremental information. Quite a bit of the type-3 record is
    calculated from different structures under current, so that may be another
    performance issue although hopefully not since, in the vast majority of
    context switches, we're just adding a simple check to see if too much time
    has elapsed since the last type-6. Only every N hours would more of a load
    be put on the switch.

    Anyway, that's where I'm at for now. I'd appreciate any comments from those
    almost certainly more knowledgable than I.

    Two strings walk into a bar. The first one says: "Hello, I'd like some
    Vodka please.ytewsr@)W$(#*$&!^Y@)^&30@#!".
    "You'll have to excuse my friend," the second one says, "he's not
    Allan Chandler (Pax), Senior IT Specialist, Australia Development
    Laboratory (ADL), Perth, IBM Corporation
    Internet:, Phone: +61 8 926 13019, Fax: +61 8 921
    Location code MP25, 77 St George's Terrace (Allendale Square), Level 25,
    Perth WA 6000

     \ /
      Last update: 2009-01-13 07:55    [W:0.041 / U:47.660 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site