lkml.org 
[lkml]   [2015]   [Sep]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patches in this message
/
Date
From
SubjectRe: [PATCH] perf tools: session: avoid infinite loop
Em Fri, Sep 18, 2015 at 02:37:09PM +0100, Mark Rutland escreveu:
> > > So it looks like I shouldn't have any synthesized events. Have I missed
> > > anything?
> >
> > Yes, you are right. But you are not getting the COMM and MMAP events from
> > the exec which means you are killing perf before it execs the workload.
>
> Oh, I see.
>
> > Perf writes through a pipe to its forked child to do the exec, so
> > to reproduce it you just need to put everything on the same cpu and play around
> > with the sleep number
> >
> > taskset -c 0 tools/perf/perf record -- sleep 1 & sleep 0.005 ; kill -2 $!
> >
> > reproduces the problem for me. Of course, since the workload doesn't get exec'ed
> > it doesn't matter if it is bogus i.e.
> >
> > taskset -c 0 tools/perf/perf record -- sdfgsdgdg & sleep 0.005 ; kill -2 $!
> >
> > also reproduces the problem.
>
> Thanks for confirming!

I tried what you suggested, but couldn't reproduce it, I only managed
when I applied the patch below _and_ executed;

# echo 1 > /proc/sys/kernel/kptr_restrict

To be on the same page as Mark. Then, yes, I get stuck in:

#0 0x00007f746a1d8aba in mmap64 () at ../sysdeps/unix/syscall-template.S:81
#1 0x00000000004be66e in __perf_session__process_events (file_size=232, data_size=<optimized out>, data_offset=<optimized out>, session=0x1ba8590)
at util/session.c:1604
#2 perf_session__process_events (session=0x1ba8590) at util/session.c:1685
#3 0x000000000042d97b in process_buildids (rec=0x869ac0 <record>) at builtin-record.c:364
#4 __cmd_record (rec=0x869ac0 <record>, argv=<optimized out>, argc=<optimized out>) at builtin-record.c:734
#5 cmd_record (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin-record.c:1199
#6 0x0000000000479123 in run_builtin (p=p@entry=0x874a90 <commands+144>, argc=argc@entry=3, argv=argv@entry=0x7ffd92eb5ee0) at perf.c:370
#7 0x0000000000420aba in handle_internal_command (argv=0x7ffd92eb5ee0, argc=3) at perf.c:429
#8 run_argv (argv=0x7ffd92eb5c70, argcp=0x7ffd92eb5c7c) at perf.c:473
#9 main (argc=3, argv=0x7ffd92eb5ee0) at perf.c:588
(gdb)

After applying your patch it works, had to tweak the commit log to avoid
starting lines with ---, breaks git scripts, also added a commiter log,
check it, patch is below, after the one I used to not process any
samples.

- Arnaldo

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 142eeb341b29..b32bb9814b35 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -645,7 +645,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
}

auxtrace_snapshot_enabled = 1;
- for (;;) {
+ for (;child_finished=1,0;) {
int hits = rec->samples;

if (record__mmap_read_all(rec) < 0) {
----
commit dd486ec4aa33cfca2fd912ef501d49909005de79
Author: Mark Rutland <mark.rutland@arm.com>
Date: Wed Sep 16 18:18:49 2015 +0100

perf record: Avoid infinite loop at buildid processing with no samples

If a session contains no events, we can get stuck in an infinite loop in
__perf_session__process_events, with a non-zero file_size and data_offset, but
a zero data_size.

In this case, we can mmap the entirety of the file (consisting of the file and
attribute headers), and fetch_mmaped_event will correctly refuse to read any
(unmapped and non-existent) event headers. This causes
__perf_session__process_events to unmap the file and retry with the exact same
parameters, getting stuck in an infinite loop.

This has been observed to result in an exit-time hang when counting
rare/unschedulable events with perf record, and can be triggered artificially
with the script below:

----
#!/bin/sh
printf "REPRO: launching perf\n";
./perf record -e software/config=9/ sleep 1 &
PERF_PID=$!;
sleep 0.002;
kill -2 $PERF_PID;
printf "REPRO: waiting for perf (%d) to exit...\n" "$PERF_PID";
wait $PERF_PID;
printf "REPRO: perf exited\n";
----
To avoid this, have __perf_session__process_events bail out early when
the file has no data (i.e. it has no events).

Commiter note:

I only managed to reproduce this when setting
/proc/sys/kernel/kptr_restrict to '1' and changing the code to
purposefully not process any samples and no synthesized samples, i.e.
kptr_restrict prevents 'record' from synthesizing the kernel mmaps for
vmlinux + modules and since it is a workload started from perf, we don't
synthesize mmap/comm records for existing threads.

Adrian Hunter managed to reproduce it in his environment tho.

Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1442423929-12253-1-git-send-email-mark.rutland@arm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 8a4537ee9bc3..fc3f7c922f99 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1580,7 +1580,10 @@ static int __perf_session__process_events(struct perf_session *session,
file_offset = page_offset;
head = data_offset - page_offset;

- if (data_size && (data_offset + data_size < file_size))
+ if (data_size == 0)
+ goto out;
+
+ if (data_offset + data_size < file_size)
file_size = data_offset + data_size;

ui_progress__init(&prog, file_size, "Processing events...");

\
 
 \ /
  Last update: 2015-09-18 17:21    [W:0.330 / U:0.004 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site