lkml.org 
[lkml]   [1998]   [Jun]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subject2.1.105 MASSIVE memory leak while stress testing
Hello Folks,

I was stress testing 2.1.105 with all of Bill Hawes's recent patches,
when the machine suddenly became ever more unresponsive.

Scenario:
1) runx -bpp 32 (KDE w. 4 terms active)
2) mount laptop (486/40+2.0.32) via knfs on 2.1.105 server, and start
kernel build.
3) start 1gig tcp throughput test to laptop (repeat when finished)
4) start glibc build in server
5) start make -j bzImage on server (repeat when finished)

All went well until 11:45 when I noticed that the glibc build and
make -j bzImage weren't making any more progress, and X became VERY
slow to respond. This kept getting worse as time went on. The kernel
build from laptop kept running fine as did the tcp throughput test.

I let it continue to grind until 13:30 before killing everything via
an rlogin from the laptop. I then ran a Bonnie -s 100 to flush the
system, and discovered that I had 64Mb still tied up afterward, whereas
normally it's only 6-7Mb. I took it down to single user and repeated
the Bonnie to no effect.

After regaining control:
Bootup: Mon Jun 8 06:58:49 1998 Load average: 27.32 41.04 43.17 2/58 18045

user : 1:28:07.69 23.1% page in : 2928447 disk 1: 419r 0w
nice : 0:00:00.00 0.0% page out: 573381 disk 2: 811277r 164126w
system: 2:01:29.48 31.9% swap in : 1614484 disk 3: 861799r 165369w
idle : 2:51:12.07 45.0% swap out: 523871 disk 4: 672828r 14446w
uptime: 6:20:49.22 context : 14338770

irq 0: 2284924 timer irq 8: 1 rtc
irq 1: 4309 keyboard irq 9: 0
irq 2: 0 cascade [4] irq 10: 5880840 NE2000
irq 3: 0 irq 11: 2689543 aic7xxx
irq 4: 0 irq 12: 146584 PS/2 Mouse
irq 5: 0 soundblaster irq 13: 1 fpu
irq 6: 0 irq 14: 0
irq 7: 0 irq 15: 18310 HiSax

Patches:
arch386_ldt104-patch buffer_104-patch char_select103-patch
fork_files103-patch mmap_104-patch n_tty103-patch
net_frag103-patch net_tcp103-patch nfsd_104-patch
pipe_103-patch swap_103-patch swap_vm104-patch
fpu_memcpy patch-ikd-2.1.103

Syslog entries are attached. The last Sys-Rq-M is after the attempt
to purge memory with Bonnie.

I've built a memleak enabled kernel and will try to reproduce this.

-Mike

Jun 8 07:02:47 mikeg kernel: tcp_do_sendmsg: limiting extra 2700 to 846
Jun 8 07:02:47 mikeg kernel: tcp_do_sendmsg: limiting extra 2700 to 846
Jun 8 07:02:47 mikeg kernel: tcp_do_sendmsg: limiting extra 2596 to 742
Jun 8 07:02:47 mikeg kernel: tcp_do_sendmsg: limiting extra 2908 to 1054
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 2700 to 846
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 3280 to 1426
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 2584 to 730
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 3044 to 1190
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 2940 to 1086
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 3400 to 1546
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 2140 to 286
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 2156 to 302
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 3180 to 1326
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 1920 to 66
Jun 8 07:02:48 mikeg kernel: tcp_do_sendmsg: limiting extra 3360 to 1506
Jun 8 07:02:49 mikeg kernel: tcp_do_sendmsg: limiting extra 2700 to 846
Jun 8 07:02:49 mikeg last message repeated 3 times
Jun 8 07:02:49 mikeg kernel: tcp_do_sendmsg: limiting extra 3012 to 1158
Jun 8 07:02:49 mikeg kernel: tcp_do_sendmsg: limiting extra 1900 to 46
Jun 8 07:02:49 mikeg kernel: tcp_do_sendmsg: limiting extra 2776 to 922
Jun 8 07:02:49 mikeg kernel: tcp_do_sendmsg: limiting extra 2080 to 226
Jun 8 07:02:49 mikeg kernel: tcp_do_sendmsg: limiting extra 3520 to 1666
Jun 8 07:20:36 mikeg kernel: SysRq: Show Memory
Jun 8 07:20:36 mikeg kernel: Mem-info:
Jun 8 07:20:36 mikeg kernel: Free pages: 1852kB
Jun 8 07:20:36 mikeg kernel: ( 31*4kB 22*8kB 5*16kB 26*32kB 8*64kB 1*128kB = 1852kB)
Jun 8 07:20:36 mikeg kernel: Swap cache: add 9855/9855, delete 9644/9644, find 0/0
Jun 8 07:20:36 mikeg kernel: Free swap: 250688kB
Jun 8 07:20:36 mikeg kernel: 20480 pages of RAM
Jun 8 07:20:36 mikeg kernel: 1240 reserved pages
Jun 8 07:20:36 mikeg kernel: 4405 pages shared
Jun 8 07:20:36 mikeg kernel: 211 pages swap cached
Jun 8 07:20:36 mikeg kernel: Buffer memory: 4092kB
Jun 8 07:20:36 mikeg kernel: Buffer heads: 1896
Jun 8 07:20:36 mikeg kernel: Buffer blocks: 1860
Jun 8 07:20:36 mikeg kernel: CLEAN: 1563 buffers, 37 used (last=37), 0 locked, 0 protected, 0 dirty
Jun 8 07:20:36 mikeg kernel: LOCKED: 239 buffers, 31 used (last=229), 0 locked, 0 protected, 0 dirty
Jun 8 07:20:36 mikeg kernel: Networking buffers in use : 11
Jun 8 07:20:36 mikeg kernel: Total network buffer allocations : 73017
Jun 8 07:20:36 mikeg kernel: Total failed network buffer allocs : 0
Jun 8 07:20:36 mikeg kernel: IP fragment buffer size : 0
Jun 8 11:45:32 mikeg kernel: SysRq: Show Memory
Jun 8 11:45:33 mikeg kernel: Mem-info:
Jun 8 11:45:33 mikeg kernel: Free pages: 1632kB
Jun 8 11:45:33 mikeg kernel: ( 38*4kB 131*8kB 7*16kB 6*32kB 0*64kB 1*128kB = 1632kB)
Jun 8 11:45:33 mikeg kernel: Swap cache: add 1389754/1389754, delete 1389106/1389106, find 0/0
Jun 8 11:45:33 mikeg kernel: Free swap: 140384kB
Jun 8 11:45:33 mikeg kernel: 20480 pages of RAM
Jun 8 11:45:33 mikeg kernel: 1240 reserved pages
Jun 8 11:45:33 mikeg kernel: 384 pages shared
Jun 8 11:45:33 mikeg kernel: 648 pages swap cached
Jun 8 11:45:33 mikeg kernel: Buffer memory: 416kB
Jun 8 11:45:33 mikeg kernel: Buffer heads: 281
Jun 8 11:45:33 mikeg kernel: Buffer blocks: 209
Jun 8 11:45:33 mikeg kernel: CLEAN: 152 buffers, 20 used (last=20), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:33 mikeg kernel: LOCKED: 52 buffers, 48 used (last=48), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:33 mikeg kernel: DIRTY: 1 buffers, 0 used (last=0), 0 locked, 0 protected, 1 dirty
Jun 8 11:45:33 mikeg kernel: Networking buffers in use : 14
Jun 8 11:45:33 mikeg kernel: Total network buffer allocations : 7863458
Jun 8 11:45:33 mikeg kernel: Total failed network buffer allocs : 0
Jun 8 11:45:33 mikeg kernel: IP fragment buffer size : 0
Jun 8 11:45:33 mikeg kernel: SysRq: Show Memory
Jun 8 11:45:33 mikeg kernel: Mem-info:
Jun 8 11:45:33 mikeg kernel: Free pages: 1612kB
Jun 8 11:45:33 mikeg kernel: ( 17*4kB 131*8kB 11*16kB 6*32kB 0*64kB 1*128kB = 1612kB)
Jun 8 11:45:33 mikeg kernel: Swap cache: add 1389781/1389781, delete 1389113/1389113, find 0/0
Jun 8 11:45:33 mikeg kernel: Free swap: 140396kB
Jun 8 11:45:33 mikeg kernel: 20480 pages of RAM
Jun 8 11:45:33 mikeg kernel: 1240 reserved pages
Jun 8 11:45:33 mikeg kernel: 409 pages shared
Jun 8 11:45:33 mikeg kernel: 668 pages swap cached
Jun 8 11:45:33 mikeg kernel: Buffer memory: 416kB
Jun 8 11:45:33 mikeg kernel: Buffer heads: 281
Jun 8 11:45:33 mikeg kernel: Buffer blocks: 209
Jun 8 11:45:33 mikeg kernel: CLEAN: 152 buffers, 20 used (last=20), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:33 mikeg kernel: LOCKED: 52 buffers, 48 used (last=48), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:33 mikeg kernel: DIRTY: 1 buffers, 0 used (last=0), 0 locked, 0 protected, 1 dirty
Jun 8 11:45:33 mikeg kernel: Networking buffers in use : 13
Jun 8 11:45:33 mikeg kernel: Total network buffer allocations : 7863603
Jun 8 11:45:33 mikeg kernel: Total failed network buffer allocs : 0
Jun 8 11:45:33 mikeg kernel: IP fragment buffer size : 0
Jun 8 11:45:55 mikeg kernel: SysRq: Show Memory
Jun 8 11:45:56 mikeg kernel: Mem-info:
Jun 8 11:45:56 mikeg kernel: Free pages: 1692kB
Jun 8 11:45:56 mikeg kernel: ( 57*4kB 129*8kB 7*16kB 6*32kB 0*64kB 1*128kB = 1692kB)
Jun 8 11:45:56 mikeg kernel: Swap cache: add 1392579/1392579, delete 1391942/1391942, find 0/0
Jun 8 11:45:56 mikeg kernel: Free swap: 140400kB
Jun 8 11:45:56 mikeg kernel: 20480 pages of RAM
Jun 8 11:45:56 mikeg kernel: 1240 reserved pages
Jun 8 11:45:56 mikeg kernel: 436 pages shared
Jun 8 11:45:56 mikeg kernel: 637 pages swap cached
Jun 8 11:45:56 mikeg kernel: Buffer memory: 404kB
Jun 8 11:45:56 mikeg kernel: Buffer heads: 277
Jun 8 11:45:56 mikeg kernel: Buffer blocks: 206
Jun 8 11:45:56 mikeg kernel: CLEAN: 131 buffers, 20 used (last=20), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:56 mikeg kernel: LOCKED: 53 buffers, 48 used (last=49), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:56 mikeg kernel: DIRTY: 1 buffers, 0 used (last=0), 0 locked, 0 protected, 1 dirty
Jun 8 11:45:56 mikeg kernel: Networking buffers in use : 27
Jun 8 11:45:56 mikeg kernel: Total network buffer allocations : 7874229
Jun 8 11:45:56 mikeg kernel: Total failed network buffer allocs : 0
Jun 8 11:45:56 mikeg kernel: IP fragment buffer size : 0
Jun 8 11:45:56 mikeg kernel: SysRq: Show Memory
Jun 8 11:45:56 mikeg kernel: Mem-info:
Jun 8 11:45:56 mikeg kernel: Free pages: 1572kB
Jun 8 11:45:56 mikeg kernel: ( 27*4kB 129*8kB 7*16kB 6*32kB 0*64kB 1*128kB = 1572kB)
Jun 8 11:45:56 mikeg kernel: Swap cache: add 1392609/1392609, delete 1391943/1391943, find 0/0
Jun 8 11:45:56 mikeg kernel: Free swap: 140404kB
Jun 8 11:45:56 mikeg kernel: 20480 pages of RAM
Jun 8 11:45:56 mikeg kernel: 1240 reserved pages
Jun 8 11:45:56 mikeg kernel: 458 pages shared
Jun 8 11:45:56 mikeg kernel: 666 pages swap cached
Jun 8 11:45:56 mikeg kernel: Buffer memory: 404kB
Jun 8 11:45:56 mikeg kernel: Buffer heads: 278
Jun 8 11:45:56 mikeg kernel: Buffer blocks: 206
Jun 8 11:45:56 mikeg kernel: CLEAN: 131 buffers, 20 used (last=20), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:56 mikeg kernel: LOCKED: 53 buffers, 48 used (last=49), 0 locked, 0 protected, 0 dirty
Jun 8 11:45:56 mikeg kernel: DIRTY: 1 buffers, 0 used (last=0), 0 locked, 0 protected, 1 dirty
Jun 8 11:45:56 mikeg kernel: Networking buffers in use : 27
Jun 8 11:45:56 mikeg kernel: Total network buffer allocations : 7874414
Jun 8 11:45:56 mikeg kernel: Total failed network buffer allocs : 0
Jun 8 11:45:56 mikeg kernel: IP fragment buffer size : 0
Jun 8 13:34:01 mikeg kernel: SysRq: Show Memory
Jun 8 13:34:01 mikeg kernel: Mem-info:
Jun 8 13:34:01 mikeg kernel: Free pages: 10488kB
Jun 8 13:34:01 mikeg kernel: ( 1576*4kB 357*8kB 59*16kB 8*32kB 0*64kB 1*128kB = 10488kB)
Jun 8 13:34:01 mikeg kernel: Swap cache: add 2172298/2172298, delete 2172280/2172280, find 0/0
Jun 8 13:34:01 mikeg kernel: Free swap: 258508kB
Jun 8 13:34:01 mikeg kernel: 20480 pages of RAM
Jun 8 13:34:01 mikeg kernel: 1240 reserved pages
Jun 8 13:34:01 mikeg kernel: 170 pages shared
Jun 8 13:34:01 mikeg kernel: 18 pages swap cached
Jun 8 13:34:01 mikeg kernel: Buffer memory: 4156kB
Jun 8 13:34:01 mikeg kernel: Buffer heads: 1163
Jun 8 13:34:01 mikeg kernel: Buffer blocks: 1135
Jun 8 13:34:01 mikeg kernel: CLEAN: 1035 buffers, 20 used (last=20), 0 locked, 0 protected, 0 dirty
Jun 8 13:34:01 mikeg kernel: LOCKED: 64 buffers, 48 used (last=59), 0 locked, 0 protected, 0 dirty
Jun 8 13:34:01 mikeg kernel: DIRTY: 1 buffers, 0 used (last=0), 0 locked, 0 protected, 1 dirty
Jun 8 13:34:01 mikeg kernel: Networking buffers in use : 0
Jun 8 13:34:01 mikeg kernel: Total network buffer allocations : 8588926
Jun 8 13:34:01 mikeg kernel: Total failed network buffer allocs : 0
Jun 8 13:34:01 mikeg kernel: IP fragment buffer size : 0
\
 
 \ /
  Last update: 2005-03-22 13:43    [from the cache]
©2003-2011 Jasper Spaans