lkml.org 
[lkml]   [2012]   [Mar]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    Subjectextreme system load [kswapd]
    From
    Hello @All,

    We have a problem on our production machine with high CPU utilization
    caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
    of SWAP.


    # cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 5.6 (Tikanga)
    #


    # uptime
    09:52:13 up 38 days, 17:01, 14 users, load average: 43.93, 50.02, 51.36
    #


    # free -m
    total used free shared buffers cached
    Mem: 128989 75577 53412 0 416 57131
    -/+ buffers/cache: 18029 110960
    Swap: 81919 31310 50609
    #

    ### from TOP:

    Tasks: 1590 total, 3 running, 1578 sleeping, 1 stopped, 8 zombie
    Cpu(s): 1.1%us, 1.9%sy, 0.0%ni, 74.4%id, 22.4%wa, 0.0%hi, 0.2%si, 0.0%st
    Mem: 132085528k total, 77397544k used, 54687984k free, 426472k buffers
    Swap: 83886064k total, 32043168k used, 51842896k free, 58475692k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    1816 root 10 -5 0 0 0 D 203.0 0.0 1124:39 kswapd3
    10801 root 15 0 113m 23m 1680 S 158.6 0.0 1694:13 vxconfigd
    3925 root RT 0 162m 15m 3368 S 140.1 0.0 1313:32 multipathd
    21399 wsy5067 16 0 22768 3192 1704 S 50.3 0.0 0:12.96 top
    10583 root 15 0 0 0 0 S 43.8 0.0 58:56.55 dmp_daemon
    16673 root 16 0 17720 1220 792 S 37.3 0.0 732:23.49 cmafcad
    24636 oracle 15 0 3314m 41m 40m S 26.6 0.0 29:31.84 oracle
    487 oracle 15 0 3301m 90m 88m S 21.4 0.1 1:05.68 oracle
    14197 root 18 0 315m 15m 3984 S 21.1 0.0 346:15.70 vxsvc
    26071 root 15 0 23900 6788 2092 S 19.8 0.0 12:31.44 MountAgent
    16418 root 15 0 5308 3656 592 S 19.1 0.0 544:54.78 cmahostd
    485 oracle 15 0 3301m 87m 87m S 18.8 0.1 1:03.92 oracle
    21456 root 15 0 0 0 0 S 18.5 0.0 14:16.09 pdflush
    4699 root 22 0 65204 8420 1568 S 15.6 0.0 137:15.75 vxdclid
    16603 root 18 0 8820 6808 332 D 14.3 0.0 1290:32 cmaperfd
    22047 tlmcron1 18 0 1227m 115m 6812 S 12.3 0.1 554:55.86 java
    6157 oracle 16 0 3302m 118m 115m D 11.4 0.1 0:01.29 oracle
    30669 oracle 16 0 2066m 141m 134m D 10.7 0.1 0:20.19 oracle
    357 tlmtst3 18 0 101m 15m 9292 S 9.7 0.0 0:29.43 stlfetch
    15005 oracle 16 0 2046m 84m 75m D 9.7 0.1 0:01.75 oracle
    32744 tlmtst3 18 0 101m 15m 9292 S 8.8 0.0 0:31.52 stlfetch
    26077 root 15 0 12116 4228 1936 S 7.8 0.0 4:26.71 VolumeAgent
    17955 oracle 16 0 3301m 28m 25m D 7.1 0.0 0:00.22 oracle
    26076 root 15 0 23268 5284 2224 S 6.8 0.0 4:00.50 OracleAgent
    111 root 10 -5 0 0 0 S 6.5 0.0 4:57.32 events/13
    5112 oracle 16 0 3314m 35m 35m S 6.5 0.0 10:38.55 oracle
    10440 oracle 16 0 3314m 36m 36m S 6.5 0.0 10:54.65 oracle
    26074 root 15 0 23012 4904 2136 S 6.5 0.0 3:09.46 NetlsnrAgent
    8631 oracle 16 0 3315m 37m 36m S 6.2 0.0 12:04.57 oracle
    26636 oracle 15 0 3316m 40m 39m S 6.2 0.0 13:07.75 oracle
    10780 oracle 16 0 6386m 39m 39m D 5.8 0.0 10:57.01 oracle
    13091 tlmtst5 15 0 423m 18m 13m S 5.8 0.0 0:02.76 gedit
    17212 oracle 15 0 2050m 39m 39m S 5.8 0.0 11:30.82 oracle
    20283 oracle 15 0 3314m 41m 40m S 5.8 0.0 10:49.00 oracle
    13147 oracle 15 0 3314m 43m 43m S 5.5 0.0 10:43.12 oracle
    16875 oracle 15 0 3316m 42m 40m S 5.5 0.0 22:52.66 oracle
    28143 root 15 0 111m 6700 3396 S 5.5 0.0 143:25.46 opcmona
    5718 oracle 16 0 3314m 38m 36m S 5.2 0.0 11:30.95 oracle
    22163 oracle 15 0 3316m 39m 38m S 5.2 0.0 32:51.94 oracle
    26065 root 15 0 22960 4724 2164 S 4.5 0.0 1:53.23 ApplicationAgen
    26612 oracle 18 0 3302m 22m 19m S 4.2 0.0 23:31.48 oracle
    8597 oracle 18 0 3302m 18m 16m S 3.9 0.0 23:30.62 oracle
    10690 oracle 18 0 6373m 20m 16m S 3.9 0.0 23:28.74 oracle
    11822 oracle 18 0 2038m 29m 25m S 3.9 0.0 0:05.17 oracle

    Why the kswapd daemon causing so high CPU utilization? Can it be
    caused by some process which try to access memory pages which are not
    in memory, but in swap? Or can it be cause by swapping for exapmple
    some inactive oracle database SGA from memory to swap?

    Thank you for help.


    Regards,

    Karol Sebesta


    \
     
     \ /
      Last update: 2012-03-20 10:11    [W:0.024 / U:32.352 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site