lkml.org 
[lkml]   [2004]   [Apr]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    SubjectStrange 'zombie' problem both in 2.4 and 2.6
    Date
    Hello.

    Some time ago I was faced with a strange problem in 2.4 kernel.
    I could reproduce in only on one system - a production 2-CPU server that is
    used as LTSP server here and also runs tons of services and MUST be always
    up.

    The problem is the following.
    Server runs normally (and uptime may be already several weeks, but may be
    only several hours).
    Suddenly something happens.
    And process table becomes full of zombies.
    Looks like any thread created by any program becomes a zombie when
    finished. Same programs (actually, same running processes) join()ed
    finished threads correctly before Something Happened. So it looks very
    much that Something happens inside the kernel.
    Affected programs include mozilla, clamav, mysqld, licq and anything else
    that creates short-living threads, or at least threads that live shorter
    than program itself.

    It looks like at some moment kernel looses the abitily to inform process
    that their threads are over. AKAIK, this is done by SIGCHLD? Anyway,
    manual sending SIGCHLD to the parent of zombies does not help.

    After the problem happens, server becomes unusable (because of process
    table overflow) in several minutes. One time Something Else happened, and
    all those zombies disappeared. In all other cases a reboot was required.

    If the process that created those "zombie thread" is terminated (i.e.
    sevice stopped), all his zombies disappear. However, after service is
    restarted, zombies become to appear again.

    Athough I tried, I could not find any correlation between making system to
    this "zombie-keeping" state and anything else happenning with the system.
    Looks like that running java apps (with blackdown jdk) makes this happen
    more often, bot still no direct correlation.

    The problem happened with official 2.4.23, 2.4.24 and 2.4.25 kernels,
    compiled from kernel.org sources.

    Yestedray I was tired with this zombie problem (it arised twice during this
    week), and decided to upgrade server to kernel 2.6.
    I installed 2.6.4 kernel from the Debain kernel-image-2.6.4-1-k7-smp
    package.

    Unfortunately, this did not eliminate the problem: it happened today again.
    The difference is that when running in 2.6, most binaries use NPTL libs
    from /lib/i686/cmov/, and seem not to be affected by the problem (i.e. no
    zombies from them). However, users need to run some statically-linked
    binaries (without source available) that have non-NPTL libs statically
    linked and so still use linuxthreads; those are affected (i.e. do create
    zombies). So problem is not rendering server unusable (so it no longer
    that critical), but it still exists in the 2.6 kernel.

    I can't reproduce the problem on any other host. And the affected system is
    a production server that is somewhat difficult to use for debugging :(
    It is a dual-K7 server with Tyan Tiger MPX S2466 motherboard and 2 Gb of
    ram. Output of 'lspci -vv' and 'cat /proc/cpuinfo' is attached. I may
    provide any other technical information.

    I'm a seasoned unix developer and sysadmin, and have some kernel hacking
    experience. However, I don't work with the kernel currently, so I am not
    "in context of" kernel internals.
    So I'm looking either for a fix :), or for some advice on what to do with
    this (i.e. where to look in the kernel code and what to look for).

    Nikita Youshchenko,
    sysadmin at lvk.cs.msu.su
    > uname -a
    Linux zigzag 2.6.4-1-k7-smp #1 SMP Sun Mar 14 00:19:02 EST 2004 i686 GNU/Linux

    > cat /proc/cpuinfo
    processor : 0
    vendor_id : AuthenticAMD
    cpu family : 6
    model : 10
    model name : AMD Athlon(tm) MP 2800+
    stepping : 0
    cpu MHz : 2133.583
    cache size : 512 KB
    fdiv_bug : no
    hlt_bug : no
    f00f_bug : no
    coma_bug : no
    fpu : yes
    fpu_exception : yes
    cpuid level : 1
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
    bogomips : 4210.68

    processor : 1
    vendor_id : AuthenticAMD
    cpu family : 6
    model : 10
    model name : AMD Athlon(tm) Processor
    stepping : 0
    cpu MHz : 2133.583
    cache size : 512 KB
    fdiv_bug : no
    hlt_bug : no
    f00f_bug : no
    coma_bug : no
    fpu : yes
    fpu_exception : yes
    cpuid level : 1
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
    bogomips : 4259.84

    > lspci -vv
    00:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] System Controller (rev 20)
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
    Latency: 32
    Region 0: Memory at f4000000 (32-bit, prefetchable) [size=64M]
    Region 1: Memory at f0600000 (32-bit, prefetchable) [size=4K]
    Region 2: I/O ports at 1020 [disabled] [size=4]
    Capabilities: <available only to root>

    00:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] AGP Bridge (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 99
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
    I/O behind bridge: 00002000-00002fff
    Memory behind bridge: f0100000-f01fffff
    Prefetchable memory behind bridge: f8000000-fbffffff
    BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B-

    00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ISA (rev 05)
    Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 0

    00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-768 [Opus] IDE (rev 04) (prog-if 8a [Master SecP PriP])
    Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] IDE
    Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 0
    Region 4: I/O ports at f000 [size=16]

    00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ACPI (rev 03)
    Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] ACPI
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-

    00:08.0 RAID bus controller: Promise Technology, Inc. PDC20271 (rev 02) (prog-if 85)
    Subsystem: Promise Technology, Inc.: Unknown device 4d68
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 64 (1000ns min, 4500ns max), Cache Line Size: 0x10 (64 bytes)
    Interrupt: pin A routed to IRQ 20
    Region 0: I/O ports at 1038 [size=8]
    Region 1: I/O ports at 1030 [size=4]
    Region 2: I/O ports at 1028 [size=8]
    Region 3: I/O ports at 1024 [size=4]
    Region 4: I/O ports at 1010 [size=16]
    Region 5: Memory at f0000000 (32-bit, non-prefetchable) [size=64K]
    Expansion ROM at <unassigned> [disabled] [size=64K]
    Capabilities: <available only to root>

    00:10.0 PCI bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] PCI (rev 05) (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
    Latency: 64
    Bus: primary=00, secondary=02, subordinate=02, sec-latency=168
    I/O behind bridge: 00003000-00003fff
    Memory behind bridge: f0200000-f03fffff
    BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-

    01:05.0 VGA compatible controller: ATI Technologies Inc Rage 128 RF/SG AGP (prog-if 00 [VGA])
    Subsystem: ATI Technologies Inc Magnum/Xpert128/X99/Xpert2000
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B+
    Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 66 (2000ns min), Cache Line Size: 0x10 (64 bytes)
    Interrupt: pin A routed to IRQ 18
    Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
    Region 1: I/O ports at 2000 [size=256]
    Region 2: Memory at f0100000 (32-bit, non-prefetchable) [size=16K]
    Expansion ROM at <unassigned> [disabled] [size=128K]
    Capabilities: <available only to root>

    02:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-768 [Opus] USB (rev 07) (prog-if 10 [OHCI])
    Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] USB
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR+
    Latency: 64 (20000ns max)
    Interrupt: pin D routed to IRQ 19
    Region 0: Memory at f0300000 (32-bit, non-prefetchable) [size=4K]

    02:04.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
    Subsystem: Intel Corp. EtherExpress PRO/100+ Management Adapter
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 64 (2000ns min, 14000ns max), Cache Line Size: 0x10 (64 bytes)
    Interrupt: pin A routed to IRQ 16
    Region 0: Memory at f0301000 (32-bit, non-prefetchable) [size=4K]
    Region 1: I/O ports at 3480 [size=64]
    Region 2: Memory at f0200000 (32-bit, non-prefetchable) [size=1M]
    Expansion ROM at <unassigned> [disabled] [size=1M]
    Capabilities: <available only to root>

    02:06.0 SCSI storage controller: Tekram Technology Co.,Ltd. TRM-S1040 (rev 01)
    Subsystem: Tekram Technology Co.,Ltd. TRM-S1040
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 64, Cache Line Size: 0x10 (64 bytes)
    Interrupt: pin A routed to IRQ 18
    Region 0: I/O ports at 3000 [size=256]
    Region 1: Memory at f0302000 (32-bit, non-prefetchable) [size=4K]
    Expansion ROM at <unassigned> [disabled] [size=64K]
    Capabilities: <available only to root>

    02:07.0 Multimedia audio controller: Ensoniq 5880 AudioPCI (rev 02)
    Subsystem: Ensoniq Creative Sound Blaster AudioPCI128
    Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort+ <MAbort- >SERR- <PERR-
    Latency: 64 (3000ns min, 32000ns max)
    Interrupt: pin A routed to IRQ 19
    Region 0: I/O ports at 34c0 [size=64]
    Capabilities: <available only to root>

    02:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)
    Subsystem: Tyan Computer: Unknown device 2466
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 80 (2500ns min, 2500ns max), Cache Line Size: 0x10 (64 bytes)
    Interrupt: pin A routed to IRQ 19
    Region 0: I/O ports at 3400 [size=128]
    Region 1: Memory at f0303000 (32-bit, non-prefetchable) [size=128]
    Expansion ROM at <unassigned> [disabled] [size=128K]
    Capabilities: <available only to root>

    \
     
     \ /
      Last update: 2005-03-22 14:02    [W:0.035 / U:1.792 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site