Messages in this thread | | | Date | Wed, 01 Jun 2011 21:37:48 +0200 | From | Gilles Hamel <> | Subject | Re: iotop: khugepaged at 99.99% (2.6.38.3) |
| |
On 12/05/2011 16:03, Andrea Arcangeli wrote: > On Wed, May 11, 2011 at 10:53:18AM +0000, Ulrich Keller wrote: >> I am seeing exactly the same symptoms on my Lenovo T60 Core2 duo, 3GB RAM, >> running Arch Linux i686 with Kernel 2.6.38.6. When I've heavily used Firefox for >> a while, or used R with high memory usage (>1 GB), individual applications >> become unresponsive, new processes fail to start and after a while the whole >> system freezes. When it happens, iotop shows khugepaged and sometimes firefox at >> 99.99%. > SYSRQ+T run multiple times during the hang and /proc/zoneinfo as well > run multiple times during the hang is the best info we can have for > now, /proc/zoneinfo is the most interesting as it will show us the > values that the too_many_isolated loop is checking to decide if to > continue looping. Me too :( Since running 2.6.38, it was happening only 3 times on the same process (convert from the ImageMagick toolkit). The last time I'm running 2.6.38.7. This process is launched every 15 minutes by crond, like this :
*/15 * * * * convert -delay 50 http://www.meteo60.org/radars/radar-nord-picardie-idf{-90,-90,-90,-75,-60,-45,-30,-15,,,}.png -loop 0 $HOME/temp/radar-pluie.gif >/dev/null 2>&1 */15 * * * * convert -delay 50 http://www.sat24.com/image.ashx\?ok=1\&country=fr\&type=slide\&time=\&index={9,9,9,8,7,6,5,4,3,2,1,1,1}\&sat=vis -loop 0 $HOME/temp/radar-nuage.gif >/dev/null 2>&1
When it's happen, I'm using firefox. Here, the whole system continues functioning normally a moment, then the X server hangs. Only these 3 tasks was stuck at 99% io busy in iotop :
$ iotop -ob Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 26 be/7 root 0.00 B/s 0.00 B/s 0.00 % 96.84 % [khugepaged] 22839 be/4 hamelg 0.00 B/s 0.00 B/s 0.00 % 96.84 % convert -delay 50 http://www.meteo60.... 22841 be/4 hamelg 0.00 B/s 0.00 B/s 0.00 % 96.84 % convert -delay 50 http://www.sat24.... Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 22839 be/4 hamelg 0.00 B/s 0.00 B/s 0.00 % 99.99 % convert -delay 50 http://www.meteo60... 26 be/7 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [khugepaged] 22841 be/4 hamelg 0.00 B/s 0.00 B/s 0.00 % 99.99 % convert -delay 50 http://www.sat24... ...
Before rebooting, I followed your hints. You'll find the output of multiple SYSRQ+T, /proc/zoneinfo and ps axu and my config.gz :
http://gilles.hamel.free.fr/config.gz http://gilles.hamel.free.fr/typescript http://gilles.hamel.free.fr/sysrq+t.txt
I hope these additional clues will help you to hunt this bug.
Regards
| |