Messages in this thread | | | From | Kalin KOZHUHAROV <> | Subject | Re: Help track a memory leak in 2.6.0..14 | Date | Thu, 15 Dec 2005 12:44:22 +0900 |
| |
Arjan van de Ven wrote: >>Who uses that memory? > > > the programs > > slabtop > xrestop > cat /proc/meminfo > > are probably good starting points to find the leak; they give a detailed > overview of various memory pools
Thank you for the info, I didn't know about xrestop :-)
So here is a present state of the machine: ss ~ # date Thu Dec 15 12:19:16 JST 2005 ss ~ # free -m total used free shared buffers cached Mem: 233 228 5 0 64 8 -/+ buffers/cache: 155 78 Swap: 0 0 0
ss ~ # cat /proc/meminfo MemTotal: 239460 kB MemFree: 5588 kB Buffers: 66212 kB Cached: 8920 kB SwapCached: 0 kB Active: 53708 kB Inactive: 28328 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 239460 kB LowFree: 5588 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 9752 kB Slab: 148528 kB CommitLimit: 119728 kB Committed_AS: 16316 kB PageTables: 260 kB VmallocTotal: 794324 kB VmallocUsed: 2276 kB VmallocChunk: 791880 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 4096 kB
ss ~ # ps -A v k-size PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 5153 tty2 Ss+ 0:01 10 515 4384 2044 0.8 -bash 4140 ? Ss 0:00 298 247 4724 860 0.3 /usr/sbin/cupsd 6572 pts/0 Ss 0:01 2 515 4124 2832 1.1 -bash 6720 pts/0 R+ 0:00 0 55 2676 760 0.3 ps -A v k-size 50 3448 ? Ss 0:00 17 133 1894 1044 0.4 /usr/sbin/cannaserver 6567 ? Ss 0:00 6 292 6087 2112 0.8 sshd: root@pts/0 4044 ? Ss 0:00 4 319 1796 736 0.3 /sbin/dhclient -cf /etc/dhcp/dhclient.co 4680 ? Ss 0:00 60 21 2146 500 0.2 login -- kalin 4485 ? Ss 0:00 3 292 3103 404 0.1 /usr/sbin/sshd 3381 ? Ss 0:00 28 100 1747 564 0.2 /usr/sbin/syslog-ng 4363 ? Ss 0:00 0 33 1694 120 0.0 /sbin/rpc.statd 4681 tty3 Ss+ 0:00 0 11 1452 64 0.0 /sbin/agetty 38400 tty3 linux 4704 tty4 Ss+ 0:00 0 11 1452 64 0.0 /sbin/agetty 38400 tty4 linux 4709 tty5 Ss+ 0:00 0 11 1452 60 0.0 /sbin/agetty 38400 tty5 linux 6015 tty1 Ss+ 0:00 1 11 1452 520 0.2 /sbin/agetty 38400 tty1 linux 4288 ? Ss 0:00 0 10 1681 116 0.0 /sbin/portmap 4714 tty6 Ss+ 0:00 0 11 1448 60 0.0 /sbin/agetty 38400 tty6 linux 841 ? S<s 0:00 1 22 1445 212 0.0 udevd 1 ? S 0:00 345 22 1441 160 0.0 init [3] 4215 ? Ss 0:07 739 16 2083 212 0.0 /usr/sbin/fnfxd 4550 ? Ss 0:00 186 9 1454 284 0.1 /usr/sbin/uptimed 2 ? SN 0:00 0 0 0 0 0.0 [ksoftirqd/0] 3 ? S< 0:00 0 0 0 0 0.0 [events/0] 4 ? S< 0:00 0 0 0 0 0.0 [khelper] 5 ? S< 0:00 0 0 0 0 0.0 [kthread] 7 ? S< 0:47 0 0 0 0 0.0 [kacpid] 95 ? S< 0:00 0 0 0 0 0.0 [kblockd/0] 134 ? S 0:00 0 0 0 0 0.0 [pdflush] 137 ? S< 0:00 0 0 0 0 0.0 [aio/0] 136 ? S 0:01 0 0 0 0 0.0 [kswapd0] 724 ? S< 0:00 0 0 0 0 0.0 [kseriod] 753 ? S< 0:00 0 0 0 0 0.0 [reiserfs/0] 1245 ? S< 0:00 0 0 0 0 0.0 [khubd] 1410 ? S 0:00 0 0 0 0 0.0 [pccardd] 1477 ? S 0:00 0 0 0 0 0.0 [pccardd] 4399 ? S< 0:01 0 0 0 0 0.0 [rpciod/0] 4400 ? S 0:00 0 0 0 0 0.0 [lockd] 5330 ? S 0:00 0 0 0 0 0.0 [pdflush]
According to slabtop: Active / Total Objects (% used) : 1806688 / 1826914 (98.9%) Active / Total Slabs (% used) : 36958 / 36958 (100.0%) Active / Total Caches (% used) : 64 / 106 (60.4%) Active / Total Size (% used) : 138777.10K / 143195.01K (96.9%) Minimum / Average / Maximum Object : 0.02K / 0.08K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 1768936 1768884 99% 0.07K 34018 52 136072K size-64 19642 16616 84% 0.06K 322 61 1288K buffer_head 10152 4828 47% 0.14K 376 27 1504K dentry_cache
ss ~ # cat /proc/slabinfo |grep "size-64 " size-64 1767932 1768000 76 52 1 : tunables 32 16 0 : slabdata 33999 34000 0 : globalstat 21513120 1767964 34005 0 0 1 84 0 : cpustat 614583639 1370079 612951759 1234047
ss ~ # lsmod Module Size Used by nfs 114540 1 lockd 66920 2 nfs nfs_acl 3680 1 nfs sunrpc 152004 4 nfs,lockd,nfs_acl usbhid 27876 0 yenta_socket 26028 2 rsrc_nonstatic 14112 1 yenta_socket ehci_hcd 33224 0 ohci_hcd 21988 0 usbcore 125660 4 usbhid,ehci_hcd,ohci_hcd e100 37984 0 mii 5568 1 e100
OK, now stop whatever I can, unload most modules and again:
ss ~ # free -m total used free shared buffers cached Mem: 233 227 6 0 63 10 -/+ buffers/cache: 153 80 Swap: 0 0 0
ss ~ # ps -A v k-size PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 5153 tty2 Ss+ 0:01 10 515 4384 2044 0.8 -bash 6572 pts/0 Ss 0:01 2 515 4124 2840 1.1 -bash 7352 pts/0 R+ 0:00 0 55 2676 756 0.3 ps -A v k-size 6567 ? Ss 0:00 6 292 6087 2112 0.8 sshd: root@pts/0 4044 ? Ss 0:00 4 319 1796 736 0.3 /sbin/dhclient -cf /etc/dhcp/dhclient.co 4680 ? Ss 0:00 60 21 2146 500 0.2 login -- kalin 4485 ? Ss 0:00 3 292 3103 404 0.1 /usr/sbin/sshd 3381 ? Ss 0:00 28 100 1747 564 0.2 /usr/sbin/syslog-ng 4681 tty3 Ss+ 0:00 0 11 1452 64 0.0 /sbin/agetty 38400 tty3 linux 4704 tty4 Ss+ 0:00 0 11 1452 64 0.0 /sbin/agetty 38400 tty4 linux 4709 tty5 Ss+ 0:00 0 11 1452 60 0.0 /sbin/agetty 38400 tty5 linux 6015 tty1 Ss+ 0:00 1 11 1452 520 0.2 /sbin/agetty 38400 tty1 linux 4714 tty6 Ss+ 0:00 0 11 1448 60 0.0 /sbin/agetty 38400 tty6 linux 841 ? S<s 0:00 1 22 1445 256 0.1 udevd 1 ? S 0:00 345 22 1441 160 0.0 init [3] 4550 ? Ss 0:00 186 9 1454 284 0.1 /usr/sbin/uptimed 2 ? SN 0:00 0 0 0 0 0.0 [ksoftirqd/0] 3 ? S< 0:00 0 0 0 0 0.0 [events/0] 4 ? S< 0:00 0 0 0 0 0.0 [khelper] 5 ? S< 0:00 0 0 0 0 0.0 [kthread] 7 ? S< 0:47 0 0 0 0 0.0 [kacpid] 95 ? S< 0:00 0 0 0 0 0.0 [kblockd/0] 134 ? S 0:00 0 0 0 0 0.0 [pdflush] 137 ? S< 0:00 0 0 0 0 0.0 [aio/0] 136 ? S 0:01 0 0 0 0 0.0 [kswapd0] 724 ? S< 0:00 0 0 0 0 0.0 [kseriod] 753 ? S< 0:00 0 0 0 0 0.0 [reiserfs/0] 1245 ? S< 0:00 0 0 0 0 0.0 [khubd] 1410 ? S 0:00 0 0 0 0 0.0 [pccardd] 1477 ? S 0:00 0 0 0 0 0.0 [pccardd] 5330 ? S 0:00 0 0 0 0 0.0 [pdflush]
ss ~ # cat /proc/slabinfo |grep "size-64 " size-64 1770831 1770860 76 52 1 : tunables 32 16 0 : slabdata 34055 34055 0 : globalstat 21551153 1770860 34060 0 0 1 84 0 : cpustat 615607009 1372499 613972465 1236227
ss ~ # lsmod Module Size Used by yenta_socket 26028 2 rsrc_nonstatic 14112 1 yenta_socket usbcore 125660 1 e100 37984 0 mii 5568 1 e100
Now, I cannot `kill -9 1410` or 1477 (the pccard) or unload yeanta_socket and rsrc_nonstatic (that never worked).
ss ~ # reboot
Now let's how things changed... From 153MB we are back down to 14MB in use... that is a leak of more than half my pshysical RAM...
ss ~ # free -m total used free shared buffers cached Mem: 233 29 204 0 3 11 -/+ buffers/cache: 14 219 Swap: 0 0 0
ss ~ # cat /proc/meminfo MemTotal: 239460 kB MemFree: 209544 kB Buffers: 3324 kB Cached: 12276 kB SwapCached: 0 kB Active: 14448 kB Inactive: 5720 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 239460 kB LowFree: 209544 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 7996 kB Slab: 6564 kB CommitLimit: 119728 kB Committed_AS: 13456 kB PageTables: 252 kB VmallocTotal: 794324 kB VmallocUsed: 2276 kB VmallocChunk: 791880 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 4096 kB
ss ~ # lsmod Module Size Used by nfs 114540 1 lockd 66920 2 nfs nfs_acl 3680 1 nfs sunrpc 152004 4 nfs,lockd,nfs_acl usbhid 27876 0 yenta_socket 26028 2 rsrc_nonstatic 14112 1 yenta_socket ehci_hcd 33224 0 ohci_hcd 21988 0 usbcore 125660 4 usbhid,ehci_hcd,ohci_hcd e100 37984 0 mii 5568 1 e100 ss ~ # cat /proc/slabinfo |grep "size-64 " size-64 1264 1300 76 52 1 : tunables 32 16 0 : slabdata 25 25 0 : globalstat 6357 1280 31 1 0 1 84 0 : cpustat 151935 427 150804 317 ss ~ # ps -A v k-size PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 4065 ? Ss 0:00 10 247 4724 1872 0.7 /usr/sbin/cupsd 4651 pts/0 Ss 0:00 8 515 3992 2804 1.1 -bash 4678 pts/0 R+ 0:00 0 55 2676 760 0.3 ps -A v k-size 3969 ? Ss 0:00 0 319 1796 996 0.4 /sbin/dhclient -cf /etc/dhcp/dhclient.co 4646 ? Ss 0:00 2 292 5927 2060 0.8 sshd: root@pts/0 4410 ? Ss 0:00 0 292 3103 1508 0.6 /usr/sbin/sshd 3373 ? Ss 0:00 0 133 1634 764 0.3 /usr/sbin/cannaserver 3306 ? Ss 0:00 0 100 1747 880 0.3 /usr/sbin/syslog-ng 4288 ? Ss 0:00 0 33 1698 824 0.3 /sbin/rpc.statd 4213 ? Ss 0:00 0 10 1685 648 0.2 /sbin/portmap 4619 tty3 Ss+ 0:00 0 11 1452 528 0.2 /sbin/agetty 38400 tty3 linux 4621 tty5 Ss+ 0:00 0 11 1452 528 0.2 /sbin/agetty 38400 tty5 linux 801 ? S<s 0:00 0 22 1449 480 0.2 udevd 4614 tty1 Ss+ 0:00 0 11 1448 524 0.2 /sbin/agetty 38400 tty1 linux 4618 tty2 Ss+ 0:00 0 11 1448 520 0.2 /sbin/agetty 38400 tty2 linux 4620 tty4 Ss+ 0:00 0 11 1448 524 0.2 /sbin/agetty 38400 tty4 linux 4622 tty6 Ss+ 0:00 0 11 1448 524 0.2 /sbin/agetty 38400 tty6 linux 1 ? S 0:00 16 22 1441 504 0.2 init [3] 4140 ? Ss 0:00 0 16 2083 560 0.2 /usr/sbin/fnfxd 4475 ? Ss 0:00 0 9 1450 508 0.2 /usr/sbin/uptimed 2 ? SN 0:00 0 0 0 0 0.0 [ksoftirqd/0] 3 ? S< 0:00 0 0 0 0 0.0 [events/0] 4 ? S< 0:00 0 0 0 0 0.0 [khelper] 5 ? S< 0:00 0 0 0 0 0.0 [kthread] 7 ? S< 0:00 0 0 0 0 0.0 [kacpid] 95 ? S< 0:00 0 0 0 0 0.0 [kblockd/0] 134 ? S 0:00 0 0 0 0 0.0 [pdflush] 135 ? S 0:00 0 0 0 0 0.0 [pdflush] 137 ? S< 0:00 0 0 0 0 0.0 [aio/0] 136 ? S 0:00 0 0 0 0 0.0 [kswapd0] 724 ? S< 0:00 0 0 0 0 0.0 [kseriod] 753 ? S< 0:00 0 0 0 0 0.0 [reiserfs/0] 1243 ? S< 0:00 0 0 0 0 0.0 [khubd] 1411 ? S 0:00 0 0 0 0 0.0 [pccardd] 1490 ? S 0:00 0 0 0 0 0.0 [pccardd] 4324 ? S< 0:00 0 0 0 0 0.0 [rpciod/0] 4325 ? S 0:00 0 0 0 0 0.0 [lockd]
ss ~ # uname -a Linux ss 2.6.13.3-ss #1 Tue Oct 18 14:08:05 Local time zone must be set--see zic manu i686 Intel(R) Pentium(R) III Mobile CPU 800MHz GenuineIntel GNU/Linux
** Hmm, have to check my timezone :-) **
So, given all this info, does anybody see anything strange? What else can I do? Or, I should put kdb in action... What else can I compare before and after a reboot? Will /proc/config.gz provide any info (just trying to save bandwith, can post it anytime)?
This is a vanilla 2.6.13.3 + mppe-mppc-1.3.patch (as module), but the patch/module was never used/loaded for this session.
NB: In this situation, X was shut down, so xrestop is irrelevant.
Thank you, Kalin. -- |[ ~~~~~~~~~~~~~~~~~~~~~~ ]| +-> http://ThinRope.net/ <-+ |[ ______________________ ]|
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |