Messages in this thread | | | Date | Tue, 15 Sep 1998 21:32:30 -0700 (PDT) | From | Simon Kirby <> | Subject | Very bad swap bug -- 2.0, 2.1 at least |
| |
This swap bug that I mentioned a while back is still happening, and this time seems to be much worse than before. In this particular case it is happening on a medium-loaded web server running 2.0.35. It's causing the load to go up to 5 or so due to the disk I/O being so saturated that many user processes get blocked.
"vmstat 1" output:
procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 1 0 0 5376 18236 10200 52244 380 0 174 0 292 305 2 9 89 1 3 0 5376 18148 10200 52380 444 0 292 0 365 387 17 17 65 1 1 0 5376 17508 10200 52668 404 0 410 0 374 369 9 15 77 2 0 0 5376 17464 10200 52644 112 0 28 0 227 149 9 12 80 4 1 0 5376 16912 10200 52796 24 0 166 0 242 163 10 8 83 0 2 0 5376 16620 10200 52944 480 0 276 0 353 365 7 13 80 1 0 0 5376 16684 10200 53008 436 0 205 0 336 377 26 20 53 0 1 0 5376 17232 10200 53084 348 0 164 0 270 265 8 12 81 2 0 0 5376 16932 10200 53220 124 0 183 0 220 173 6 15 80 5 3 0 5376 16576 10200 53300 420 0 186 0 285 297 14 8 78 2 2 0 5376 16516 10200 53300 500 0 127 0 324 355 7 10 83 2 1 0 5376 15928 10200 53320 520 0 147 0 295 355 7 13 80 1 0 0 5376 16308 10200 53348 180 0 89 0 256 282 34 13 53 1 1 0 5376 15852 10200 53524 240 0 245 0 260 242 10 13 78 0 0 0 5376 16028 10200 53596 264 0 125 0 275 471 13 9 79
As you can see, the amount of stuff swapped out is staying the same, but it is constantly "swapping in" something. There is obviously a lot of memory available to have this data swapped back in and out of swap space, but it isn't happening.
From the tests I did before, it seems what is being "swapped in" is the executable code of a program which has been previously swapped out. I had a program running that sleep()ed for a few seconds and then ran a small main loop (stat()ed a file) -- every time the main loop ran, vmstat would show the swap in ("si") number as non-zero. I confirmed this by stracing the program and running "vmstat 1" at the same time.
Issuing a "swapoff -a ; swapon -a" causes the program to go away, until something else forces some programs out to swap. (In this case, it seems to be some customer's run-away CGI script).
This is quite a bad problem -- this going unnoticed for a long time would kill a drive after a while. It has happened several times now here. When I first noticed it, it was happening on several machines, too. (luckily only in small amounts, though).
Anybody at least know what causes this?
Simon-
| Simon Kirby | Systems Administration | | mailto:sim@netnation.com | NetNation Communications | | http://www.netnation.com/ | Tech: (604) 684-6892 |
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |