Messages in this thread | | | From | Zoltán Kiss Z <> | Subject | [perf] periodic performance monitoring interrupts causing heavy delays | Date | Tue, 21 Jan 2020 13:28:53 +0100 |
| |
Hi,
I'm running some benchmarking on a Dell Poweredge R630 server (Xeon E5-2650 v4 Broadwell) with Ubuntu 18.04 (4.15.0-62 kernel). The application is busy looping on core's reserved with isolcpus. However every 5 seconds there is a ~1 ms glitch where my cores seem to be stopped. After checking interrupts it seems these glitches coincide with an NMI and a "Performance monitoring interrupt" on all the busy looping cores, while the idle ones doesn't receive it. I was not running perf during these occasions, but the below post suggests that there is still some kernel perf activity in the background which causes these interrupts:
https://serverfault.com/questions/714648/how-to-disable-perf-subsystem-in-linux-kernel
I can also see these "perf interrupt took too long" messages even when I never started perf by myself since the last reboot. I've tried to adjust the /proc/sys/kernel/perf_* knobs, without any luck.
The above post suggests that these periodic interrupts cannot be disabled in any way, and I would like to figure out if this is really the case, and if yes, why? A 1 ms glitch in every 5 seconds is quite unacceptable not just because of benchmarking noise, but also for latency sensitive applications like mine. Also, I couldn't find any info about these periodic perf background jobs generating the interrupts, could anyone point me to at least some function names where I could start digging, so I could verify if indeed these glitches are due to perf?
Thanks,
Zoltan Kiss
| |