Messages in this thread | | | From | Thomas Habets <> | Subject | PROBLEM: Memory leak in -test9? | Date | Tue, 11 Nov 2003 16:36:57 +0100 |
| |
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
(I'm not on the list. CC any replies)
[2.] There seems to be a memory leak in 2.6.0-test9. Before I go on I'll mention that the box is running Debian sarge (testing), and the compiler is gcc 3.3.1.
A lot of memory is used (and eventually it will crash), but very little is in cache or buffers. After shutting down zebra, postgresql, snmp and apache this is what the system looked like. Notice the Slab-size in /proc/meminfo. I don't know what slab is, but it's getting bigger... I think. It's not that big after reboot anyway. The memory gets used up after a few days of uptime, and when all memory is used, the box eventually becomes unresponsive due to excessive swap use. Note that this box doesn't do anything (almost). It's a workstation that's not in use. It has the above mentioned daemons running (but unused) and also sshs every 5 minutes to another box for mrtg graph stats. (rsa keys)
/proc/config.gz attached.
/proc/sysvipc/{sem,shm,msg} are empty.
I'll leave it running in case you want more info, but it will fill up all mem and crash in a few days if it does what some other -test? did.
Sorry if it just looks like it will do the same, and this is all just normal behavior.
[3.] Memory leak, slab
[4.] Linux version 2.6.0-test9 (andy@rex) (gcc version 3.3.1 20030626 (Debian prerelease)) #4 Tue Oct 28 15:12:18 CET 2003
[7.1] $ sh scripts/ver_linux If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes. Linux rex 2.6.0-test9 #4 Tue Oct 28 15:12:18 CET 2003 i686 GNU/Linux Gnu C 3.3.1 Gnu make 3.80 util-linux 2.11z mount 2.11z module-init-tools 0.9.13-pre e2fsprogs 1.34-WIP nfs-utils 1.0.3 Linux C Library 2.3.1 Dynamic linker (ldd) 2.3.1 Procps 3.1.9 Net-tools 1.60 Console-tools 0.2.3 Sh-utils 5.0 Modules Loaded twofish serpent aes blowfish sha256 dummy 8139too mii crc32
[7.2.] Processor information (from /proc/cpuinfo): $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 6 model name : Celeron (Mendocino) stepping : 5 cpu MHz : 467.898 cache size : 128 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr bogomips : 923.64
[7.3.] Module information (from /proc/modules): $ cat /proc/modules twofish 38752 0 - Live 0xcc89d000 serpent 13600 0 - Live 0xcc880000 aes 31616 0 - Live 0xcc886000 blowfish 9824 0 - Live 0xcc87c000 sha256 13888 0 - Live 0xcc877000 dummy 1924 0 - Live 0xcc869000 8139too 24320 0 - Live 0xcc870000 mii 5152 1 8139too, Live 0xcc864000 crc32 4320 1 8139too, Live 0xcc861000
[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem) $ cat /proc/ioports /proc/iomem 0000-001f : dma1 0020-0021 : pic1 0040-005f : timer 0060-006f : keyboard 0070-0077 : rtc 0080-008f : dma page reg 00a0-00a1 : pic2 00c0-00df : dma2 00f0-00ff : fpu 0170-0177 : ide1 01f0-01f7 : ide0 0213-0213 : ISAPnP 0376-0376 : ide1 03c0-03df : vga+ 03f6-03f6 : ide0 0a79-0a79 : isapnp write 0cf8-0cff : PCI conf1 4000-403f : 0000:00:07.3 5000-501f : 0000:00:07.3 d000-dfff : PCI Bus #01 e000-e01f : 0000:00:07.2 e000-e01f : uhci_hcd e400-e4ff : 0000:00:08.0 e800-e8ff : 0000:00:0b.0 e800-e8ff : 8139too f000-f00f : 0000:00:07.1 f000-f007 : ide0 f008-f00f : ide1 00000000-0009fbff : System RAM 0009fc00-0009ffff : reserved 000a0000-000bffff : Video RAM area 000c0000-000c7fff : Video ROM 000f0000-000fffff : System ROM 00100000-0bfeffff : System RAM 00100000-00490e26 : Kernel code 00490e27-005a08bf : Kernel data 0bff0000-0bff2fff : ACPI Non-volatile Storage 0bff3000-0bffffff : ACPI Tables e0000000-e3ffffff : 0000:00:00.0 e4000000-e5ffffff : PCI Bus #01 e4000000-e4ffffff : 0000:01:00.0 e6000000-e6ffffff : PCI Bus #01 e6000000-e6ffffff : 0000:01:00.0 e7000000-e7ffffff : 0000:00:08.0 ea000000-ea0000ff : 0000:00:0b.0 ea000000-ea0000ff : 8139too ea001000-ea001fff : 0000:00:08.0 ffff0000-ffffffff : reserved
[7.5.] PCI information ('lspci -vvv' as root) That's a lot, and I doubt it would help. Ask if you need it.
[7.6.] SCSI information (from /proc/scsi/scsi) No SCSI
[7.7.] Other information that might be relevant to the problem (please look in /proc and include all information that you think to be relevant):
$ uptime 00:51:38 up 6 days, 11:54, 5 users, load average: 0.02, 0.02, 0.00
$ free -m total used free shared buffers cached Mem: 184 168 15 0 7 12 -/+ buffers/cache: 148 35 Swap: 243 74 168
(these two commands were run a day after the above mentioned daemons were shut off: RSS $ ps uax | awk '{ foo+= $6 } END{print foo}' 36380 $ free -m total used free shared buffers cached Mem: 184 168 16 0 3 11 - -/+ buffers/cache: 153 31 Swap: 243 87 155 now back to yesterdays /proc/meminfo)
$ cat /proc/meminfo MemTotal: 189240 kB MemFree: 16208 kB Buffers: 7536 kB Cached: 12940 kB SwapCached: 18400 kB Active: 22316 kB Inactive: 27176 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 189240 kB LowFree: 16208 kB SwapTotal: 248996 kB SwapFree: 172572 kB Dirty: 380 kB Writeback: 0 kB Mapped: 24460 kB Slab: 121272 kB Committed_AS: 89412 kB PageTables: 744 kB VmallocTotal: 843720 kB VmallocUsed: 604 kB VmallocChunk: 843048 kB
$ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 1212 428 ? S Nov04 0:04 init [2] root 2 0.0 0.0 0 0 ? SWN Nov04 0:00 [ksoftirqd/0] root 3 0.0 0.0 0 0 ? SW< Nov04 0:01 [events/0] root 4 0.0 0.0 0 0 ? SW< Nov04 0:04 [kblockd/0] root 5 0.0 0.0 0 0 ? SW Nov04 0:00 [khubd] root 6 0.0 0.0 0 0 ? SW Nov04 0:00 [kapmd] root 8 0.0 0.0 0 0 ? SW Nov04 0:01 [pdflush] root 9 0.0 0.0 0 0 ? SW Nov04 0:09 [kswapd0] root 10 0.0 0.0 0 0 ? SW< Nov04 0:00 [aio/0] root 11 0.0 0.0 0 0 ? SW< Nov04 0:00 [xfslogd/0] root 12 0.0 0.0 0 0 ? SW< Nov04 0:00 [xfsdatad/0] root 13 0.0 0.0 0 0 ? SW Nov04 0:00 [pagebufd] root 15 0.0 0.0 0 0 ? SW Nov04 0:00 [kseriod] root 16 0.0 0.0 0 0 ? SW Nov04 0:30 [kjournald] root 30 0.0 0.5 2568 964 ? S Nov04 0:00 /sbin/devfsd /dev root 178 0.0 0.2 1408 424 ? S Nov04 0:00 dhclient -pf /var/run/dhclient.eth0.pid eth0 daemon 189 0.0 0.1 1316 288 ? S Nov04 0:00 /sbin/portmap root 273 0.0 0.2 1296 508 ? S Nov04 0:28 /sbin/syslogd root 279 0.0 0.2 2456 396 ? S Nov04 0:00 /sbin/klogd root 293 0.0 0.2 1240 392 ? S Nov04 0:00 /usr/sbin/inetd root 397 0.0 0.2 2684 492 ? S Nov04 0:00 /usr/sbin/sshd root 400 0.0 0.2 1208 456 ? S Nov04 0:00 /usr/sbin/uptimed root 407 0.0 0.2 4128 488 ? S Nov04 0:00 /usr/bin/X11/xfs -daemon root 467 0.0 0.2 2476 428 ? S Nov04 0:00 /bin/bash /etc/rc2.d/S20xprint posix_sh_forced start root 469 0.0 0.2 4020 464 ? S Nov04 0:00 /usr/X11R6/bin/Xprt -ac -pn -nolisten tcp -audit 4 -fp /usr/X11R6/lib/X11/fonts/Type1,/usr/X11R6/lib/X11/fonts/100dpi,/usr/X11R6/lib root 470 0.0 0.1 1196 308 ? S Nov04 0:00 tee -a /dev/null root 471 0.0 0.1 1200 376 ? S Nov04 0:00 logger -p lpr.notice -t Xprt_64 root 489 0.0 0.2 1372 464 ? S Nov04 0:00 /sbin/rpc.statd root 493 0.0 1.0 1908 1900 ? SL Nov04 0:00 /usr/sbin/ntpd root 498 0.0 0.1 2952 340 ? S Nov04 0:03 /usr/sbin/arpwatch -N -p daemon 501 0.0 0.1 1320 220 ? S Nov04 0:00 /usr/sbin/atd root 504 0.0 0.1 1396 324 ? S Nov04 0:00 /usr/sbin/cron root 516 0.0 0.2 2936 432 ? S Nov04 0:00 /usr/bin/X11/xdm root 517 0.0 0.0 1200 176 tty1 S Nov04 0:00 /sbin/getty 38400 tty1 root 518 0.0 0.0 1200 176 tty2 S Nov04 0:00 /sbin/getty 38400 tty2 root 519 0.0 0.0 1200 176 tty3 S Nov04 0:00 /sbin/getty 38400 tty3 root 520 0.0 0.0 1200 176 tty4 S Nov04 0:00 /sbin/getty 38400 tty4 root 521 0.0 0.0 1200 176 tty5 S Nov04 0:00 /sbin/getty 38400 tty5 root 522 0.0 0.0 1200 176 tty6 S Nov04 0:00 /sbin/getty 38400 tty6 root 529 0.0 0.6 47508 1276 ? S Nov04 1:38 /usr/X11R6/bin/X vt7 -dpi 100 -nolisten tcp -auth /var/lib/xdm/authdir/authfiles/A:0-8vgmon root 531 0.0 0.1 3388 316 ? S Nov04 0:00 -:0 root 544 0.7 8.9 51300 16980 ? S Nov04 67:24 xconsole - -geometry 480x130-0-0 -notify -verbose -fn fixed -exitOnFail -file /dev/xconsole andy 563 0.0 0.2 3472 392 ? S Nov04 0:01 x-window-manager andy 624 0.0 0.0 2372 156 ? S Nov04 0:00 /usr/bin/ssh-agent x-window-manager andy 626 0.0 0.1 3472 256 ? S Nov04 0:00 x-window-manager -d :0.1 -s andy 628 0.0 0.1 2332 220 ? S Nov04 0:00 /usr/bin/Animate --window 0 --context 8 andy 629 0.0 0.1 3048 256 ? S Nov04 0:00 /usr/bin/Wharf - --window 0 --context 8 andy 631 0.0 0.0 2332 188 ? S Nov04 0:00 /usr/bin/Animate --window 0 --context 8 andy 632 0.0 0.1 3048 228 ? S Nov04 0:00 /usr/bin/Wharf - --window 0 --context 8 andy 635 0.0 0.1 3048 280 ? S Nov04 0:00 /usr/bin/Pager - --window 0 --context 8 0 0 andy 639 0.0 0.1 3048 204 ? S Nov04 0:00 /usr/bin/Pager - --window 0 --context 8 0 0 root 22089 0.0 0.1 1684 244 ? S Nov08 0:00 /USR/SBIN/CRON root 22090 0.0 0.0 2156 176 ? S Nov08 0:00 /bin/sh -c if [ -x /usr/bin/mrtg ] && [ -r /etc/mrtg.cfg ]; then /usr/bin/mrtg /etc/mrtg.cfg >> /var/log/mrtg/mrtg.log 2>&1; fi root 22091 0.0 0.1 8572 192 ? S Nov08 0:01 /usr/bin/perl - -w /usr/bin/mrtg /etc/mrtg.cfg andy 22093 0.0 0.1 2752 216 ? S Nov08 0:00 ssh xxxxxxx /home/andy/mrtg/linux_stats.pl eth0 root 25679 0.0 0.0 0 0 ? SW Nov09 0:00 [pdflush] andy 2463 0.0 0.3 4728 576 ? S Nov10 0:00 x-terminal-emulator -T XTerminal@rex andy 2467 0.0 0.3 4736 700 ? S Nov10 0:00 x-terminal-emulator -T XTerminal@rex andy 2470 0.0 0.8 3344 1548 pts/0 S Nov10 0:00 bash andy 2474 0.0 0.4 4728 912 ? S Nov10 0:00 x-terminal-emulator -T XTerminal@rex andy 2477 0.0 0.5 3368 968 pts/1 S Nov10 0:00 bash andy 2490 0.0 0.4 3344 800 pts/2 S Nov10 0:00 bash root 2518 0.0 0.3 5760 704 ? S 00:01 0:00 sshd: andy [priv] andy 2520 0.0 0.5 5780 948 ? S 00:01 0:00 sshd: andy@pts/3 andy 2521 0.0 0.9 3380 1868 pts/3 S 00:01 0:00 -bash andy 2763 0.0 0.4 2500 760 pts/3 R 00:27 0:00 ps aux
dmesg - -- Linux version 2.6.0-test9 (thomas@rex) (gcc version 3.3.1 20030626 (Debian prerelease)) #4 Tue Oct 28 15:12:18 CET 2003 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable) BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000000bff0000 (usable) BIOS-e820: 000000000bff0000 - 000000000bff3000 (ACPI NVS) BIOS-e820: 000000000bff3000 - 000000000c000000 (ACPI data) BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved) 191MB LOWMEM available. On node 0 totalpages: 49136 DMA zone: 4096 pages, LIFO batch:1 Normal zone: 45040 pages, LIFO batch:10 HighMem zone: 0 pages, LIFO batch:1 DMI 2.2 present. Building zonelist for node : 0 Kernel command line: root=/dev/hda3 ro Local APIC disabled by BIOS -- reenabling. Found and enabled local APIC! Initializing CPU#0 PID hash table entries: 1024 (order 10: 8192 bytes) Detected 467.898 MHz processor. Console: colour VGA+ 80x25 Memory: 188808k/196544k available (3651k kernel code, 7100k reserved, 1086k data, 184k init, 0k highmem) Calibrating delay loop... 923.64 BogoMIPS Security Scaffold v1.0.0 initialized Capability LSM initialized Dentry cache hash table entries: 32768 (order: 5, 131072 bytes) Inode-cache hash table entries: 16384 (order: 4, 65536 bytes) Mount-cache hash table entries: 512 (order: 0, 4096 bytes) CPU: After generic identify, caps: 0183fbff 00000000 00000000 00000000 CPU: After vendor identify, caps: 0183fbff 00000000 00000000 00000000 CPU: L1 I cache: 16K, L1 D cache: 16K CPU: L2 cache: 128K CPU: After all inits, caps: 0183fbff 00000000 00000000 00000040 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#0. CPU: Intel Celeron (Mendocino) stepping 05 Enabling fast FPU save and restore... done. Checking 'hlt' instruction... OK. POSIX conformance testing by UNIFIX enabled ExtINT on CPU#0 ESR value before enabling vector: 00000000 ESR value after enabling vector: 00000000 Using local APIC timer interrupts. calibrating APIC timer ... ..... CPU clock speed is 467.0661 MHz. ..... host bus clock speed is 66.0808 MHz. NET: Registered protocol family 16 PCI: PCI BIOS revision 2.10 entry at 0xfb2d0, last bus=1 PCI: Using configuration type 1 mtrr: v2.0 (20020519) Linux Plug and Play Support v0.97 (c) Adam Belay PnPBIOS: Scanning system for PnP BIOS support... PnPBIOS: Found PnP BIOS installation structure at 0xc00fbed0 PnPBIOS: PnP BIOS version 1.0, entry 0xf0000:0xbef8, dseg 0xf0000 PnPBIOS: 16 nodes reported by PnP BIOS; 16 recorded by driver drivers/usb/core/usb.c: registered new driver usbfs drivers/usb/core/usb.c: registered new driver hub PCI: Probing PCI hardware PCI: Probing PCI hardware (bus 00) PCI: Using IRQ router PIIX [8086/7110] at 0000:00:07.0 apm: BIOS version 1.2 Flags 0x07 (Driver version 1.16ac) ikconfig 0.7 with /proc/config* VFS: Disk quotas dquot_6.5.1 Coda Kernel/Venus communications, v6.0.0, coda@cs.cmu.edu devfs: v1.22 (20021013) Richard Gooch (rgooch@atnf.csiro.au) devfs: boot_options: 0x1 Installing knfsd (copyright (C) 1996 okir@monad.swb.de). udf: registering filesystem SGI XFS for Linux with ACLs, no debug enabled SGI XFS Quota Management subsystem Initializing Cryptographic API Limiting direct PCI/PCI transfers. isapnp: Scanning for PnP cards... isapnp: Card 'OPL3-SAX Sound Board' isapnp: 1 Plug & Play card detected total pty: 256 Unix98 ptys configured Real Time Clock Driver v1.12 Linux agpgart interface v0.100 (c) Dave Jones agpgart: Detected an Intel 440BX Chipset. agpgart: Maximum main memory to use for agp memory: 149M agpgart: AGP aperture is 64M @ 0xe0000000 [drm] Initialized tdfx 1.0.0 20010216 on minor 0 [drm] Initialized r128 2.5.0 20030725 on minor 1 [drm] Initialized radeon 1.9.0 20020828 on minor 2 [drm] Initialized mga 3.1.0 20021029 on minor 3 [drm] Initialized i810 1.4.0 20030605 on minor 4 [drm] Initialized i830 1.3.2 20021108 on minor 5 [drm] Initialized sis 1.1.0 20030826 on minor 6 Using anticipatory io scheduler Floppy drive(s): fd0 is 1.44M FDC 0 is a post-1991 82077 RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx PIIX4: IDE controller at PCI slot 0000:00:07.1 PIIX4: chipset revision 1 PIIX4: not 100% native mode: will probe irqs later ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:DMA hda: WDC AC28400R, ATA DISK drive ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 hdc: CD-ROM CDU701-Q, ATAPI CD/DVD-ROM drive ide1 at 0x170-0x177,0x376 on irq 15 hda: max request size: 128KiB hda: 16514064 sectors (8455 MB) w/512KiB Cache, CHS=16383/16/63, UDMA(33) /dev/ide/host0/bus0/target0/lun0: p1 p2 p3 end_request: I/O error, dev hdc, sector 0 hdc: ATAPI 14X CD-ROM drive, 128kB Cache, DMA Uniform CD-ROM driver Revision: 3.12 ohci_hcd: 2003 Oct 13 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI) ohci_hcd: block sizes: ed 64 td 64 drivers/usb/host/uhci-hcd.c: USB Universal Host Controller Interface driver v2.1 PCI: Found IRQ 10 for device 0000:00:07.2 PCI: Sharing IRQ 10 with 0000:00:0b.0 uhci_hcd 0000:00:07.2: UHCI Host Controller uhci_hcd 0000:00:07.2: irq 10, io base 0000e000 uhci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 1 hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected drivers/usb/core/usb.c: registered new driver audio drivers/usb/class/audio.c: v1.0.0:USB Audio Class driver drivers/usb/core/usb.c: registered new driver hiddev drivers/usb/core/usb.c: registered new driver hid drivers/usb/input/hid-core.c: v2.0:USB HID core driver drivers/usb/core/usb.c: registered new driver tiglusb drivers/usb/misc/tiglusb.c: TI-GRAPH LINK USB (aka SilverLink) driver, version 1.06 mice: PS/2 mouse device common for all mice input: PC Speaker input: ImPS/2 Generic Wheel Mouse on isa0060/serio1 serio: i8042 AUX port at 0x60,0x64 irq 12 input: AT Translated Set 2 keyboard on isa0060/serio0 serio: i8042 KBD port at 0x60,0x64 irq 1 Advanced Linux Sound Architecture Driver Version 0.9.7 (Thu Sep 25 19:16:36 2003 UTC). ALSA device list: No soundcards found. NET: Registered protocol family 2 IP: routing cache hash table of 1024 buckets, 8Kbytes TCP: Hash tables configured (established 16384 bind 32768) ip_conntrack version 2.1 (1535 buckets, 12280 max) - 300 bytes per conntrack ip_tables: (C) 2000-2002 Netfilter core team ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>. http://snowman.net/projects/ipt_recent/ arp_tables: (C) 2002 David S. Miller Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 10 IPv6 over IPv4 tunneling driver ip6_tables: (C) 2000-2002 Netfilter core team registering ipv6 mark target NET: Registered protocol family 17 NET: Registered protocol family 15 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Mounted devfs on /dev Freeing unused kernel memory: 184k freed Adding 248996k swap on /dev/hda2. Priority:-1 extents:1 EXT3 FS on hda3, internal journal 8139too Fast Ethernet driver 0.9.26 PCI: Found IRQ 10 for device 0000:00:0b.0 PCI: Sharing IRQ 10 with 0000:00:07.2 eth0: RealTek RTL8139 at 0xcc867000, 00:50:22:8d:44:97, IRQ 10 eth0: Identified 8139 chip type 'RTL-8139C' Disabled Privacy Extensions on device c0553260(lo) eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
- --------- typedef struct me_s { char name[] = { "Thomas Habets" }; char email[] = { "thomas@habets.pp.se" }; char kernel[] = { "Linux 2.4" }; char *pgpKey[] = { "http://www.habets.pp.se/pubkey.txt" }; char pgp[] = { "A8A3 D1DD 4AE0 8467 7FDE 0945 286A E90A AD48 E854" }; char coolcmd[] = { "echo '. ./_&. ./_'>_;. ./_" }; } me_t;
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux)
iD8DBQE/sQIaKGrpCq1I6FQRAiqcAKCXEAsg8CzjJcH/a30ISKXCNjb26gCcDqYP 9Tl1quW84TP1nHemixUylJE= =abMW -----END PGP SIGNATURE----- [unhandled content-type:application/x-gzip] | |