Messages in this thread |  | | Subject | ISSUE: Heavy NFS brings (SMP) machine down in 2.4.9-ac10 | From | Joachim Breuer <> | Date | 19 Sep 2001 16:15:10 +0200 |
| |
Hello!
[1.] One line summary of the problem: Heavy NFS brings (SMP) machine down in 2.4.9-ac10
[2.] Full description of the problem/report:
I was creating a largeish (12GB) tar, reading from a local disk and writing to an NFS-mounted directory. Performance was about what I expected (ca. 1MByte/s). About 15-25 Minutes into the process the NFS server appeared to become congested (network load meter of my desktop hub dropping rapidly); investigation showed the server (which could still be ssh'd into) at load 7.something with one of the nfsds and kupdated eating up all CPU they could get (and heavy disk thrashing when I went to hake a look). At this point, the 'top' I used to gather that information itself used up 50%-70% CPU. I interrupted (Ctrl-C) the tar on the client (which took some time, as could be expected). After the tar client had stopped the server did *not* recover from the aforementioned condition (load 7.x, nfsd/kupdated eating CPU, heavy disk thrashing) within 30 minutes. At that time I decided to stop the nfs server (which was impossible) and then reboot the server (which did not work cleanly, as well - only with Magic SysRq and some patience I could sync/unmount/reboot).
netstat -au showed the RecvQ of nfsd with about 600000 entries (in one case 715000), that number did not change *at all* until the reboot.
According to top, the machine used a stable 600-700k of swap during the thrashing with 3.5MB memory free and 100-150MB of memory used for cache/buffers; IMHO the thrashing seems fs-related.
I tried the same tar 2 more times, always encountering exactly these same symptoms.
No other NFS clients were active during the whole process mentioned above.
After switching to kernel 2.4.9 (no ac) on the server only (client still using 2.4.9-ac10) the same tar completed without incident.
A possibly related (or not) effect is this: While rebooting the server with the client in question active, but idle the client's load went up to around 8 without any processes using up significant amounts of CPU time (winner was 'top' at <3%), load normalised after it regained connectivity with the NFS server. The client used 2.4.9-ac10 as well.
[3.] Keywords (i.e., modules, networking, kernel): nfs, SMP, kupdated
[4.] Kernel version (from /proc/version):
2.4.9-ac10, verbatim version line not available as the server is "semi-production" and it's running 2.4.9 right now.
[5.] Output of Oops.. message with symbolic information resolved (see Kernel Mailing List FAQ, Section 1.5): N/A
[6.] A small shell script or example program which triggers the problem (if possible)
tar cvf /mnt/server/foo.tar /local/tree/about/12GB/
[7.] Environment
The filesystem the server was writing to uses reiserfs, default parameters. Client used same kernel version as server, reading from an ext2 filesystem.
Both filesystems (client and server) reside on software raid0 volumes, composed of one SCSI and one IDE ((U)DMA-something) disk each.
Both machines use RedHat 7.1 + fairly recent updates, with a custom-built as-modular-as-possibly 2.4.9-ac10 (no other patches) kernel tailored to the respective CPU types (Pentium Pro SMP on the server, Celeron Coppermine on the client).
The client's HW is a Celeron 800, GA-6OX, 256 MB SDRAM; aic7xxx, NetGear FA-310TX (tulip driver).
[7.1.] Software (add the output of the ver_linux script here) -- Versions installed: (if some fields are empty or looks -- unusual then possibly you have very old versions) Linux sol.fo.et.local 2.4.9 #1 SMP Tue Sep 18 22:46:46 CEST 2001 i686 unknown [At the time of the problem it was 2.4.9-ac10] Kernel modules 2.4.2 Gnu C 2.96 Binutils 2.10.91.0.2 Linux C Library 2.2.2 Dynamic Linker (ld.so) 2.2.2 ls: /usr/lib/libg++.so: No such file or directory Procps 2.0.7 Mount 2.11b Net-tools (2000-05-21) Kbd 0.3.3 Sh-utils 2.0
[7.2.] Processor information (from /proc/cpuinfo): processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 1 model name : Pentium Pro stepping : 7 cpu MHz : 198.950 cache size : 256 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov bogomips : 396.49
processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 1 model name : Pentium Pro stepping : 7 cpu MHz : 198.950 cache size : 256 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov bogomips : 397.31
[7.3.] Module information (from /proc/modules): nfsd 70304 8 (autoclean) lockd 50704 1 (autoclean) [nfsd] sunrpc 68208 1 (autoclean) [nfsd lockd] autofs 11456 2 (autoclean) natsemi 13184 1 (autoclean) ipchains 38368 0 (unused) st 26976 1 reiserfs 157904 1 (autoclean) sr_mod 13312 0 (autoclean) (unused) cdrom 28192 0 (autoclean) [sr_mod] raid0 3648 2 md 43424 2 [raid0] sd_mod 11360 4 aic7xxx 115440 5 scsi_mod 96080 4 [st sr_mod sd_mod aic7xxx] usb-uhci 22336 0 (unused) usbcore 53024 1 [usb-uhci] unix 17152 21 ide-disk 7104 4 ide-probe-mod 8368 0 ide-mod 157936 4 [ide-disk ide-probe-mod]
[7.4.] SCSI information (from /proc/scsi/scsi): Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: IBM Model: DNES-309170 Rev: SA30 Type: Direct-Access ANSI SCSI revision: 03 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: IBM Model: DGHS09Z Rev: 0350 Type: Direct-Access ANSI SCSI revision: 03 Host: scsi0 Channel: 00 Id: 02 Lun: 00 Vendor: IBM Model: DDRS-34560 Rev: S97B Type: Direct-Access ANSI SCSI revision: 02 Host: scsi0 Channel: 00 Id: 03 Lun: 00 Vendor: IBM Model: DPSS-309170N Rev: S96H Type: Direct-Access ANSI SCSI revision: 03 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: SONY Model: SDT-9000 Rev: 0400 Type: Sequential-Access ANSI SCSI revision: 02
[7.5.] Other information that might be relevant to the problem (please look in /proc and include all information that you think to be relevant):
Server board: Tyan 1662 (Titan Pro AT), 2x Pentium Pro 200, 256 MB EDO.
HW in the Server is all-PCI, I'll just include `lspci`: 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:07.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] (rev 01) 00:07.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:07.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:0b.0 VGA compatible controller: Matrox Graphics, Inc. MGA 2064W [Millennium] (rev 01) 00:0c.0 Ethernet controller: National Semiconductor Corporation: Unknown device 0020 [This is a NetGear FA-311 TX] 00:0d.0 SCSI storage controller: Adaptec AIC-7881U 00:0e.0 SCSI storage controller: Adaptec AHA-294x / AIC-7871
(Despite the Matrox Millenium the server is a "cellar machine", no X or other interactive use).
[X.] Other notes, patches, fixes, workarounds:
The problem certainly looks reproducible here and seems to be hidden away somewhere in the differences between 2.4.9 and 2.4.9-ac10.
As the machine is "semi-production" I did not do a lot of further testing once I saw the 2.4.9 no ac solved the problem for me, but if the bug is deemed worthy of investigation I will try to be of further assistance; I will always respond with further information about the setup if asked and will try patches etc. as far as my time permits.
So long, Joe
-- "I use emacs, which might be thought of as a thermonuclear word processor." -- Neal Stephenson, "In the beginning... was the command line" - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
|  |