lkml.org 
[lkml]   [1998]   [Sep]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectVery bad swap bug -- 2.0, 2.1 at least
    This swap bug that I mentioned a while back is still happening, and this
    time seems to be much worse than before. In this particular case it is
    happening on a medium-loaded web server running 2.0.35. It's causing the
    load to go up to 5 or so due to the disk I/O being so saturated that many
    user processes get blocked.

    "vmstat 1" output:

    procs memory swap io system cpu
    r b w swpd free buff cache si so bi bo in cs us sy id
    1 0 0 5376 18236 10200 52244 380 0 174 0 292 305 2 9 89
    1 3 0 5376 18148 10200 52380 444 0 292 0 365 387 17 17 65
    1 1 0 5376 17508 10200 52668 404 0 410 0 374 369 9 15 77
    2 0 0 5376 17464 10200 52644 112 0 28 0 227 149 9 12 80
    4 1 0 5376 16912 10200 52796 24 0 166 0 242 163 10 8 83
    0 2 0 5376 16620 10200 52944 480 0 276 0 353 365 7 13 80
    1 0 0 5376 16684 10200 53008 436 0 205 0 336 377 26 20 53
    0 1 0 5376 17232 10200 53084 348 0 164 0 270 265 8 12 81
    2 0 0 5376 16932 10200 53220 124 0 183 0 220 173 6 15 80
    5 3 0 5376 16576 10200 53300 420 0 186 0 285 297 14 8 78
    2 2 0 5376 16516 10200 53300 500 0 127 0 324 355 7 10 83
    2 1 0 5376 15928 10200 53320 520 0 147 0 295 355 7 13 80
    1 0 0 5376 16308 10200 53348 180 0 89 0 256 282 34 13 53
    1 1 0 5376 15852 10200 53524 240 0 245 0 260 242 10 13 78
    0 0 0 5376 16028 10200 53596 264 0 125 0 275 471 13 9 79

    As you can see, the amount of stuff swapped out is staying the same, but
    it is constantly "swapping in" something. There is obviously a lot of
    memory available to have this data swapped back in and out of swap space,
    but it isn't happening.

    From the tests I did before, it seems what is being "swapped in" is the
    executable code of a program which has been previously swapped out. I had
    a program running that sleep()ed for a few seconds and then ran a small
    main loop (stat()ed a file) -- every time the main loop ran, vmstat would
    show the swap in ("si") number as non-zero. I confirmed this by stracing
    the program and running "vmstat 1" at the same time.

    Issuing a "swapoff -a ; swapon -a" causes the program to go away, until
    something else forces some programs out to swap. (In this case, it seems
    to be some customer's run-away CGI script).

    This is quite a bad problem -- this going unnoticed for a long
    time would kill a drive after a while. It has happened several times now
    here. When I first noticed it, it was happening on several machines, too.
    (luckily only in small amounts, though).

    Anybody at least know what causes this?

    Simon-

    | Simon Kirby | Systems Administration |
    | mailto:sim@netnation.com | NetNation Communications |
    | http://www.netnation.com/ | Tech: (604) 684-6892 |


    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.rutgers.edu
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2005-03-22 13:44    [W:0.030 / U:63.344 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site