lkml.org 
[lkml]   [2020]   [Jan]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Subject[perf] periodic performance monitoring interrupts causing heavy delays
Date
Hi,

I'm running some benchmarking on a Dell Poweredge R630 server (Xeon
E5-2650 v4 Broadwell) with Ubuntu 18.04 (4.15.0-62 kernel). The
application is busy looping on core's reserved with isolcpus. However
every 5 seconds there is a ~1 ms glitch where my cores seem to be
stopped. After checking interrupts it seems these glitches coincide with
an NMI and a "Performance monitoring interrupt" on all the busy looping
cores, while the idle ones doesn't receive it. I was not running perf
during these occasions, but the below post suggests that there is still
some kernel perf activity in the background which causes these interrupts:

https://serverfault.com/questions/714648/how-to-disable-perf-subsystem-in-linux-kernel

I can also see these "perf interrupt took too long" messages even when I
never started perf by myself since the last reboot. I've tried to adjust
the /proc/sys/kernel/perf_* knobs, without any luck.

The above post suggests that these periodic interrupts cannot be
disabled in any way, and I would like to figure out if this is really
the case, and if yes, why? A 1 ms glitch in every 5 seconds is quite
unacceptable not just because of benchmarking noise, but also for
latency sensitive applications like mine.
Also, I couldn't find any info about these periodic perf background jobs
generating the interrupts, could anyone point me to at least some
function names where I could start digging, so I could verify if indeed
these glitches are due to perf?

Thanks,

Zoltan Kiss

\
 
 \ /
  Last update: 2020-01-21 13:29    [W:0.134 / U:0.404 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site