Messages in this thread | | | From | Carl <> | Date | Thu, 24 Dec 2009 16:12:21 +0200 | Subject | CPU eaten by unknown code |
| |
Hi.
I was benchmarking two programs and at some points i was getting completely crazy results. Upon futher investigation it was discovered that at some times the benchmarked programs would take much longer to run because "somebody" was eating CPU.
Naturally i tried to stop any other processes that could be running (also note that this is a dual-core system and the benchmarks used only the one CPU, so theoretically, even another process like cron might run on the other CPU without interrupting the benchmarks). No result.
So i wrote a reduced testcase that would exploit the problem. It's a small python script that keeps printing the time it takes to run a tripple for-loop, forever. On my system (Dual core Athlon 64, 32bit kernel+user) the standard time to run the loop is 1.2 secs. However, this is what happens:
The loop takes 1.2 seconds for about 138 seconds. The loop takes 2.4 second for about 45 seconds. And this repeats forever.
In other words, every 138 seconds something is "running" for 45 seconds and it takes 50% of my CPU time!
I have been completely unable to discover what is this thing. It is not some process as I would see it with 'ps' and 'top'. It could not be something like flushing the disc cache because 45 seconds is just too much. The network cable is completely unpluged. The thing is not visible through /proc/stat cpuload: the system load is 0% and the only way to understand this thing is eating CPU is by looking that the for-loop in the python script takes double time to run. I cannot completely exclude the possibility that I've been rootkitted although I believe this is a pretty secure system.
tested on linux 2.6.31.5 and 2.6.29.4
This is a real WTF. 45 seconds with 50% of the CPU is a *lot* of processing power for a 2.5GHz system. What could be running all this time? Anybody else seeing similar results? Ideas on how to disable this so one can run benchmarks on a really "idle" system?
The script: It prints the time to run the for-loop. It +/-5% deviation from the previous value it prints a dot. Run for at least 5 minutes on an idle system...
======================================================>8 # idle.py
import time, sys
def r (): for i in xrange (10): for i in xrange (1000): for i in xrange (1000): i = i + 1
def t (): t0 = time.time () r () return int (10000*(time.time () - t0))
l = None while 1: x = t () if l is not None: if -dt <= x - l <= dt: sys.stdout.write (".") sys.stdout.flush () else: print "(%i secs)"%(time.time () - lt) lt = time.time () print "Time to run loop:", x/10000.0 else: dt = x / 20 lt = time.time () print "Time to run loop:", x/10000.0 l = x
| |