lkml.org 
[lkml]   [2010]   [Jan]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    SubjectRe: [RFC PATCH -tip 2/2 v2] add a scripts for pagecache usage per process
    From
    Date
    Hi,

    On Fri, 2010-01-22 at 19:08 -0500, Keiichi KII wrote:
    > The scripts are implemented based on the trace stream scripting support.
    > And the scripts implement the following.
    > - how many pagecaches each process has per each file
    > - how many pages are cached per each file
    > - how many pagecaches each process shares
    >

    Nice, it looks like a very useful script - I gave it a quick try and it
    seems to work well...

    The only problem I see, nothing to do with your script and nothing you
    can do anything about at the moment, is that the record step generates a
    huge amount of data, which of course makes the event processing take
    awhile. A lot of it appears to be due to perf itself - being able to
    filter out the perf-generated events in the kernel would make a big
    difference, I think; you normally don't want to see those anyway...

    BTW, I see that you did your first version in Python - not that you'd
    want to redo it again, but just FYI I now have working Python support
    that I'll be posting soon - I still have some small details to hammer
    out, but if you have any other scripts in the pipeline, in a couple days
    you'll be able to use Python instead if you want.

    A few more small comments below...

    > To monitor pagecache usage per a process, run "pagecache-usage-record" to
    > record perf data for "pagecache-usage.pl" and run "pagecache-usage-report"
    > to display.

    Another way of course would be to use 'perf trace record/report' and the
    script name as shown by perf trace -l:

    $ perf trace record pagecache-usage
    $ perf trace report pagecache-usage

    >
    > The below outputs show execution sample.
    >
    > [file list]
    > device inode caches
    > --------------------------------
    > 253:0 1051413 130
    > 253:0 1051399 2
    > 253:0 1051414 44
    > 253:0 1051417 154
    >
    > [process list]
    > o postmaster-2330
    > cached added removed indirect
    > device inode pages pages pages removed pages
    > ----------------------------------------------------------------
    > 253:0 1051399 0 2 0 0
    > 253:0 1051417 154 0 0 0
    > 253:0 1051413 130 0 0 0
    > 253:0 1051414 44 0 0 0
    > ----------------------------------------------------------------
    > total: 337 2 0 0
    >
    > >From the output, we can know some information like:
    >
    > - if "added pages" > "cached pages" on process list then
    > It means repeating add/remove pagecache many times.
    > => Bad case for pagecache usage
    >
    > - if "added pages" <= "cached pages" on process list then
    > It means no unnecessary I/O operations.
    > => Good case for pagecache usage.
    >
    > - if "caches" on file list >
    > sum "cached pages" per each file on process list then
    > It means there are unneccessary pagecaches in the memory.
    > => Bad case for pagecache usage
    >
    > Signed-off-by: Keiichi Kii <k-keiichi@bx.jp.nec.com>
    > Cc: Atsushi Tsuji <a-tsuji@bk.jp.nec.com>
    > ---
    > tools/perf/scripts/perl/bin/pagecache-usage-record | 7
    > tools/perf/scripts/perl/bin/pagecache-usage-report | 6
    > tools/perf/scripts/perl/pagecache-usage.pl | 160 +++++++++++++++++++++
    > 3 files changed, 173 insertions(+)
    >
    > Index: linux-2.6-tip/tools/perf/scripts/perl/bin/pagecache-usage-record
    > ===================================================================
    > --- /dev/null
    > +++ linux-2.6-tip/tools/perf/scripts/perl/bin/pagecache-usage-record
    > @@ -0,0 +1,7 @@
    > +#!/bin/bash
    > +perf record -c 1 -f -a -M -R -e filemap:add_to_page_cache -e filemap:find_get_page -e filemap:remove_from_page_cache
    > +
    > +
    > +
    > +
    > +
    > Index: linux-2.6-tip/tools/perf/scripts/perl/bin/pagecache-usage-report
    > ===================================================================
    > --- /dev/null
    > +++ linux-2.6-tip/tools/perf/scripts/perl/bin/pagecache-usage-report
    > @@ -0,0 +1,6 @@
    > +#!/bin/bash
    > +# description: pagecache usage per process
    > +perf trace -s ~/libexec/perf-core/scripts/perl/pagecache-usage.pl
    > +
    > +
    > +
    > Index: linux-2.6-tip/tools/perf/scripts/perl/pagecache-usage.pl
    > ===================================================================
    > --- /dev/null
    > +++ linux-2.6-tip/tools/perf/scripts/perl/pagecache-usage.pl
    > @@ -0,0 +1,160 @@
    > +# perf trace event handlers, generated by perf trace -g perl
    You might want to get rid of this and add a short description and your
    name, if you want to take credit for it. ;-)

    > +# Licensed under the terms of the GNU GPL License version 2
    > +

    > +# The common_* event handler fields are the most useful fields common to
    > +# all events. They don't necessarily correspond to the 'common_*' fields
    > +# in the format files. Those fields not available as handler params can
    > +# be retrieved using Perl functions of the form common_*($context).
    > +# See Context.pm for the list of available functions.
    > +

    You can get rid of this part too - it's just meant to be helpful
    information generated when starting a script.

    > +use lib "$ENV{'PERF_EXEC_PATH'}/scripts/perl/Perf-Trace-Util/lib";
    > +use lib "./Perf-Trace-Util/lib";
    > +use Perf::Trace::Core;
    > +use Perf::Trace::Context;
    > +use Perf::Trace::Util;
    > +use List::Util qw/sum/;
    > +my %files;
    > +my %processes;
    > +my %records;
    > +
    > +sub trace_end
    > +{
    > + print_pagecache_usage_per_file();
    > + print "\n";
    > + print_pagecache_usage_per_process();
    > + print_unhandled();
    > +}
    > +
    > +sub filemap::remove_from_page_cache
    > +{
    > + my ($event_name, $context, $common_cpu, $common_secs, $common_nsecs,
    > + $common_pid, $common_comm,
    > + $s_dev, $i_ino, $offset) = @_;
    > + my $f = \%{$files{$s_dev}{$i_ino}};
    > + my $r = \%{$records{$common_comm."-".$common_pid}{$f}};
    > +
    > + delete $$f{$offset};
    > + $$r{inode} = $i_ino;
    > + $$r{dev} = $s_dev;
    > + if (exists $$r{added}{$offset}) {
    > + $$r{removed}++;
    > + } else {
    > + $$r{indirect_removed}++;
    > + }
    > +}
    > +
    > +sub filemap::add_to_page_cache
    > +{
    > + my ($event_name, $context, $common_cpu, $common_secs, $common_nsecs,
    > + $common_pid, $common_comm,
    > + $s_dev, $i_ino, $offset) = @_;
    > + my $f = \%{$files{$s_dev}{$i_ino}};
    > + my $r = \%{$records{$common_comm."-".$common_pid}{$f}};
    > +
    > + $$f{$offset}++;
    > + $$r{added}{$offset}++;
    > + $$r{inode} = $i_ino;
    > + $$r{dev} = $s_dev;
    > +}
    > +
    > +sub filemap::find_get_page
    > +{
    > + my ($event_name, $context, $common_cpu, $common_secs, $common_nsecs,
    > + $common_pid, $common_comm,
    > + $s_dev, $i_ino, $offset, $page) = @_;
    > + my $f = \%{$files{$s_dev}{$i_ino}};
    > + my $r = \%{$records{$common_comm."-".$common_pid}{$f}};
    > +
    > + if ($page != 0) {
    > + $$f{$offset}++;
    > + $$r{cached}++;
    > + $$r{inode} = $i_ino;
    > + $$r{dev} = $s_dev;
    > + }
    > +}
    > +
    > +my %unhandled;
    > +
    > +sub trace_unhandled
    > +{
    > + my ($event_name, $context, $common_cpu, $common_secs, $common_nsecs,
    > + $common_pid, $common_comm) = @_;
    > +
    > + $unhandled{$event_name}++;
    > +}
    > +
    > +sub print_unhandled
    > +{
    > + if ((scalar keys %unhandled) == 0) {
    > + print "unhandled events nothing\n";

    This is kind of distracting - it's not too useful to know that you don't
    have unhandled events, but if you do have some, it is useful to print
    those as you do below - it points out that some event type are being
    unnecessarily recorded or the script is being run on the wrong trace
    data.

    Thanks,

    Tom

    > + return;
    > + }
    > +
    > + print "\nunhandled events:\n\n";
    > +
    > + printf("%-40s %10s\n", "event", "count");
    > + printf("%-40s %10s\n", "----------------------------------------",
    > + "-----------");
    > +
    > + foreach my $event_name (keys %unhandled) {
    > + printf("%-40s %10d\n", $event_name, $unhandled{$event_name});
    > + }
    > +}
    > +
    > +sub minor
    > +{
    > + my $dev = shift;
    > + return $dev & ((1 << 20) - 1);
    > +}
    > +
    > +sub major
    > +{
    > + my $dev = shift;
    > + return $dev >> 20;
    > +}
    > +
    > +sub print_pagecache_usage_per_file
    > +{
    > + print "[file list]\n";
    > + printf(" %12s %10s %8s\n", "", "", "cached");
    > + printf(" %12s %10s %8s\n", "device", "inode", "pages");
    > + printf(" %s\n", '-' x 32);
    > + while(my($dev, $file) = each(%files)) {
    > + while(my($inode, $r) = each(%$file)) {
    > + my $count = values %$r;
    > + next if $count == 0;
    > + printf(" %12s %10d %8d\n",
    > + major($dev).":".minor($dev), $inode, $count);
    > + }
    > + }
    > +}
    > +
    > +sub print_pagecache_usage_per_process
    > +{
    > + print "[process list]\n";
    > + while(my ($pid, $v) = each(%records)) {
    > + my ($sum_cached, $sum_added, $sum_removed, $sum_indirect_removed);
    > +
    > + print "o $pid\n";
    > + printf(" %12s %10s %8s %8s %8s %13s\n", "", "",
    > + "cached", "added", "removed", "indirect");
    > + printf(" %12s %10s %8s %8s %8s %13s\n", "device", "inode",
    > + "pages", "pages", "pages", "removed pages");
    > + printf(" %s\n", '-' x 64);
    > + while(my ($file, $r) = each(%$v)) {
    > + my $added_num = List::Util::sum(values %{$$r{added}});
    > + $sum_cached += $$r{cached};
    > + $sum_added += $added_num;
    > + $sum_removed += $$r{removed};
    > + $sum_indirect_removed += $$r{indirect_removed};
    > + printf(" %12s %10d %8d %8d %8d %13d\n",
    > + major($$r{dev}).":".minor($$r{dev}), $$r{inode},
    > + $$r{cached}, $added_num, $$r{removed},
    > + $$r{indirect_removed});
    > + }
    > + printf(" %s\n", '-' x 64);
    > + printf(" total: %5s %10s %8d %8d %8d %13d\n", "", "", $sum_cached,
    > + $sum_added, $sum_removed, $sum_indirect_removed);
    > + print "\n";
    > + }
    > +}
    >
    >
    >



    \
     
     \ /
      Last update: 2010-01-23 09:25    [W:0.060 / U:33.176 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site