lkml.org 
[lkml]   [1998]   [Sep]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectVery bad swap bug -- 2.0, 2.1 at least
This swap bug that I mentioned a while back is still happening, and this
time seems to be much worse than before. In this particular case it is
happening on a medium-loaded web server running 2.0.35. It's causing the
load to go up to 5 or so due to the disk I/O being so saturated that many
user processes get blocked.

"vmstat 1" output:

procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 0 0 5376 18236 10200 52244 380 0 174 0 292 305 2 9 89
1 3 0 5376 18148 10200 52380 444 0 292 0 365 387 17 17 65
1 1 0 5376 17508 10200 52668 404 0 410 0 374 369 9 15 77
2 0 0 5376 17464 10200 52644 112 0 28 0 227 149 9 12 80
4 1 0 5376 16912 10200 52796 24 0 166 0 242 163 10 8 83
0 2 0 5376 16620 10200 52944 480 0 276 0 353 365 7 13 80
1 0 0 5376 16684 10200 53008 436 0 205 0 336 377 26 20 53
0 1 0 5376 17232 10200 53084 348 0 164 0 270 265 8 12 81
2 0 0 5376 16932 10200 53220 124 0 183 0 220 173 6 15 80
5 3 0 5376 16576 10200 53300 420 0 186 0 285 297 14 8 78
2 2 0 5376 16516 10200 53300 500 0 127 0 324 355 7 10 83
2 1 0 5376 15928 10200 53320 520 0 147 0 295 355 7 13 80
1 0 0 5376 16308 10200 53348 180 0 89 0 256 282 34 13 53
1 1 0 5376 15852 10200 53524 240 0 245 0 260 242 10 13 78
0 0 0 5376 16028 10200 53596 264 0 125 0 275 471 13 9 79

As you can see, the amount of stuff swapped out is staying the same, but
it is constantly "swapping in" something. There is obviously a lot of
memory available to have this data swapped back in and out of swap space,
but it isn't happening.

From the tests I did before, it seems what is being "swapped in" is the
executable code of a program which has been previously swapped out. I had
a program running that sleep()ed for a few seconds and then ran a small
main loop (stat()ed a file) -- every time the main loop ran, vmstat would
show the swap in ("si") number as non-zero. I confirmed this by stracing
the program and running "vmstat 1" at the same time.

Issuing a "swapoff -a ; swapon -a" causes the program to go away, until
something else forces some programs out to swap. (In this case, it seems
to be some customer's run-away CGI script).

This is quite a bad problem -- this going unnoticed for a long
time would kill a drive after a while. It has happened several times now
here. When I first noticed it, it was happening on several machines, too.
(luckily only in small amounts, though).

Anybody at least know what causes this?

Simon-

| Simon Kirby | Systems Administration |
| mailto:sim@netnation.com | NetNation Communications |
| http://www.netnation.com/ | Tech: (604) 684-6892 |


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:44    [W:1.118 / U:0.140 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site