Messages in this thread |  | | Date | Fri, 22 Nov 1996 10:11:22 -0600 | From | David Worden <> | Subject | 2.0.25 Frozen |
| |
Greetings,
This post ends my "lurk-only" mode for this newsgroup.
Since this is my first post, let me start out by THANKING the principle players in the Linux community for being willing to apply their skills to this effort.
WARNING: This post will be relatively long. ------------------------------------------------------------------- ----------- My background ----------------------------------------- ------------------------------------------------------------------- I am not a "kernel hacker". However, I have worked with many flavors of Unix (since 1985) in both systems administration and programming capacities. I have programmed in 'C' for several years, but have used mostly Perl for the last year and a half. My kernel knowledge comes from the usual reference works and a few classes. I am writing this only to give some context to my post. ------------------------------------------------------------------- ----------- Problem Overview/Description -------------------------- ------------------------------------------------------------------- A co-worker and I were getting ready to put a Linux machine into a production role. Before doing so, we wanted to "stress test" it a little. Here is what we did (high level):
01) Installed Slackware 96 as a baseline 02) Got latest Linux production kernel (2.0.25), built, and booted it. 03) Ran the following Perl script from "/usr/src/linux" to repeatedly build 2.0.25. one-thousand times: ------------------------------------------------------------------- #!/usr/bin/perl -w ################################################################### # Note - This script is a Perl incarnation of the script # suggested at: # http://www.bitwizard.nl/sig11/ ################################################################### $good_builds = 0;
for($x = 1; $x <= 1000; $x++) { $my_temp_file = sprintf("%s-%04.4d-TMP", $^T, $x); `make clean 1>$my_temp_file 2>&1`; if ($? == 0) { `make zImage 1>>$my_temp_file 2>&1`; if ($? == 0) { $good_builds++; unlink($my_temp_file); } } $date_time = `date`; chop($date_time); printf "[%06.6d] of [%06.6d] builds GOOD [%s]\n", $good_builds, $x, $date_time; } exit; ------------------------------------------------------------------- ----------- The Problem ------------------------------------------- ------------------------------------------------------------------- The kernel building stopped at build #72.
The machine was basically frozen [ I am in Minnesota though *8^) ].
I could switch VC's, and was able to do a "Shift-Scrollock" to show memory and a "Ctrl-Scrollock" to show state. Then, the machine totally froze.
Here are the pertinent(?) "syslog" messages (with first few fields "cut" out and some formatting): ------------------------------------------------------------------- free sibling task PC stack pid father child younger older swapper 0 R 00000000 4096 0 0 1 init -1 R current 4 1 0 16779 kflushd 2 S 00000000 4 2 1 3 kswapd 3 S 00000000 4 3 1 4 2 nfsiod 4 S 00000000 3520 4 1 5 3 nfsiod 5 S 00000000 3520 5 1 6 4 nfsiod 6 S 00000000 3520 6 1 7 5 nfsiod 7 S 00000000 3520 7 1 13 6 bash 8 S 00000000 4 74 1 1985 76 73 agetty 9 R FFFFFFFF 4 554 1 16779 79 crond 10 R 00000000 4 37 1 51 13 syslogd 11 R FFFFFFFF 4 51 1 53 37 update 12 R 00000000 4 13 1 37 7 klogd 13 R 00000000 1968 53 1 55 51 rpc.portmap 14 S FFFFFFFF 3236 55 1 57 53 inetd 15 R FFFFFFFF 1548 57 1 59 55 lpd 16 S FFFFFFFF 3376 59 1 62 57 rpc.nfsd 17 R FFFFFFFF 4 64 1 70 62 rpc.mountd 18 R FFFFFFFF 3516 62 1 64 59 sendmail 19 R 00000000 3244 70 1 73 64 script 20 S FFFFFFFF 4 1985 74 1986 gpm 21 R 001E1708 3404 73 1 74 70 agetty 22 R FFFFFFFF 4 76 1 77 74 agetty 23 S FFFFFFFF 1500 77 1 78 76 agetty 24 S FFFFFFFF 4 78 1 79 77 agetty 25 S FFFFFFFF 2772 79 1 554 78 script 26 S FFFFFFFF 4 1986 1985 1987 bash 27 S 00000000 12 1987 1986 1988 djw-test.pl 28 S 00000000 4 1988 1987 16782 djw-test.pl 29 R 00000000 4 16782 1988 cpp 37 R 00000000 1968 16779 1 554 Mem-info: Free pages: 5120kB ( 346*4kB 241*8kB 75*16kB 3*32kB 0*64kB 4*128kB = 5120kB) Swap cache: add 40/40, delete 26789394/40, find 74/0 Free swap: 200724kB 16384 pages of RAM 1294 free pages 556 reserved pages 1810 pages shared Buffer memory: 39420kB Buffer heads: 40980 Buffer blocks: 39420 Unable to handle kernel paging request at virtual address c7478124 current->tss.cr3 = 00fda000, |r3 = 00fda000 *pde = 00000000 Oops: 0000 CPU: 0 EIP: 0010:[show_buffers+137/392] EFLAGS: 00010207 eax: 0001882f ebx: 00000487 ecx: 07478110 edx: 001f7000 esi: 00000000 edi: 00005a81 ebp: 00000000 esp: 0001ce9c ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018 Process init (pid: 1, process nr: 1, stackpage=0001c000) Stack: 00004000 0000050e 0000022c 00000712 00000001 0001882f 001f7000 00000000 0000046b 0000004b 00000000 0010f8bb 00000002 0000f203 00000046 001e6ae8 001f7000 00185fa5 00185b35 00000003 00000000 0009e098 00000000 00000001 Call Trace: [show_mem+187/200] [do_spec+29/32] [handle_scancode+833/904] [timer_bh+172/332] [keyboard_interrupt+46/72] [do_IRQ+45/80] [IRQ1_interrupt+86/128] [sync_buffers+76/396] [fsync_dev+14/48] [sys_sync+7/16] [system_call+82/128] Code: 8b 41 14 a8 04 74 04 ff 44 24 28 a8 40 74 04 ff 44 24 1c a8 Aiee, killing interrupt handler -------------------------------------------------------------------
Here is the "tail" end of the build #72 temp file (slightly formatted): ------------------------------------------------------------------- gcc -D__KERNEL__ -I/usr/src/linux-2.0.25/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strength-reduce -pipe -m486 -malign-loops=2 -malign-jumps=2 -malign-functions=2 -DCPU=686 -c -o tcp.o tcp.c gcc -D__KERNEL__ -I/usr/src/linux-2.0.25/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strength-reduce -pipe -m486 -malign-loops=2 -malign-jumps=2 -malign-functions=2 -DCPU=686 -c -o tcp_input.o tcp_input.c gcc: Internal compiler error: program cc1 got fatal signal 11 make[3]: *** [tcp_input.o] Error 1 make[3]: Leaving directory `/usr/src/linux-2.0.25/net/ipv4' make[2]: *** [first_rule] Error 2 make[2]: Leaving directory `/usr/src/linux-2.0.25/net/ipv4' make[1]: *** [sub_dirs] Error 2 make[1]: Leaving directory `/usr/src/linux-2.0.25/net' make: *** [linuxsubdirs] Error 2 ------------------------------------------------------------------- ------------------------------------------------------------------- ----------- Hardware ---------------------------------------------- ------------------------------------------------------------------- The machine is a brand new Micron Millennia Pro Plus.
Memory: 63312k/65536k available (660k kernel code, 384k reserved, 1180k data) tty00 at 0x03f8 (irq = 4) is a 16550A tty01 at 0x02f8 (irq = 3) is a 16550A PS/2 auxiliary pointing device detected -- driver installed. Floppy drive(s): fd0 is 1.44M AHA-2940 Ultra (PCI-bus), I/O 0xfc00, Mem 0xffbef000: Vendor: SEAGATE Model: ST51080N Rev: 0958 Vendor: MICROP Model: 1991-27Q4D Rev: Q4D Vendor: SONY Model: SDT-5000 Rev: 3.30 Vendor: PLEXTOR Model: CD-ROM PX-8XCS Rev: 1.02 eth0: 3Com 3c595 Vortex 100baseTX at 0xff80, 00:60:97:24:40:6e, IRQ 5
[/proc/cpuinfo] processor : 0 cpu : 686 model : Pentium Pro vendor_id : GenuineIntel stepping : 7 fdiv_bug : no hlt_bug : no fpu : yes fpu_exception : yes cpuid : yes wp : yes flags : fpu vme de pse tsc msr pae mce cx8 11 mtrr pge mca cmov bogomips : 199.07
[/proc/interrupts] 0: 938188 timer 1: 1961 keyboard 2: 0 cascade 5: 33564 3c595 Vortex 100baseTX 10: 14543 + aic7xxx 12: 72 PS/2 Mouse 13: 1 math error
[/proc/ioports] 0000-001f : dma1 0020-003f : pic1 0040-005f : timer 0060-006f : keyboard 0080-009f : dma page reg 00a0-00bf : pic2 00c0-00df : dma2 00f0-00ff : npu 02f8-02ff : serial(auto) 03c0-03df : vga+ 03f0-03f5 : floppy 03f7-03f7 : floppy DIR 03f8-03ff : serial(auto) fc00-fcbe : aic7xxx ff80-ff9f : 3c595 Vortex 100baseTX
[/proc/pci] (slightly formatted) PCI devices found: Bus 0, device 17, function 0: SCSI storage controller: Adaptec AIC-7881U (rev 0). Medium devsel. Fast back-to-back capable. IRQ 10. Master Capable. Latency=64. Min Gnt=8.Max Lat=8. I/O at 0xfc00. Non-prefetchable 32 bit memory at 0xffbef000. Bus 0, device 15, function 0: Ethernet controller: 3Com 3C595 100bTX (rev 0). Medium devsel. IRQ 5. Master Capable. Latency=248. Min Gnt=3.Max Lat=8. I/O at 0xff80. Bus 0, device 11, function 0: VGA compatible controller: S3 Inc. ViRGE (rev 5). Medium devsel. IRQ 11. Master Capable. Latency=64. Min Gnt=4.Max Lat=255. Non-prefetchable 32 bit memory at 0xf8000000. Bus 0, device 7, function 1: IDE interface: Intel 82371SB Natoma/Triton II PIIX3 (rev 0). Medium devsel. Fast back-to-back capable. Master Capable. Latency=64. I/O at 0xffa0. Bus 0, device 7, function 0: ISA bridge: Intel 82371SB Natoma/Triton II PIIX3 (rev 1). Medium devsel. Fast back-to-back capable. Master Capable. No bursts. Bus 0, device 0, function 0: Host bridge: Intel 82441FX Natoma (rev 2). Medium devsel. Fast back-to-back capable. Master Capable. Latency=64. ------------------------------------------------------------------- ----------- OS/Configuration -------------------------------------- ------------------------------------------------------------------- Linux version 2.0.25 (root@spare) (gcc version 2.7.2) #1 Wed Nov 20 13:24:33 CST 1996 ------------------------------------------------------------------- Configuration (ONLY affirmative)
CONFIG_MODULES=y CONFIG_NET=y CONFIG_PCI=y CONFIG_SYSVIPC=y CONFIG_BINFMT_AOUT=y CONFIG_BINFMT_ELF=y CONFIG_KERNEL_ELF=y CONFIG_M686=y CONFIG_BLK_DEV_FD=y CONFIG_INET=y CONFIG_IP_NOSR=y CONFIG_SKB_LARGE=y CONFIG_SCSI=y CONFIG_BLK_DEV_SD=y CONFIG_CHR_DEV_ST=y CONFIG_BLK_DEV_SR=y CONFIG_SCSI_CONSTANTS=y CONFIG_SCSI_AIC7XXX=y CONFIG_NETDEVICES=y CONFIG_DUMMY=m CONFIG_NET_ETHERNET=y CONFIG_NET_VENDOR_3COM=y CONFIG_VORTEX=y CONFIG_MINIX_FS=y CONFIG_EXT2_FS=y CONFIG_FAT_FS=y CONFIG_MSDOS_FS=y CONFIG_PROC_FS=y CONFIG_NFS_FS=y CONFIG_ISO9660_FS=y CONFIG_SERIAL=y CONFIG_MOUSE=y CONFIG_PSMOUSE=y ------------------------------------------------------------------- Some other comments. We have looked at the "SIG11 problem" material located at: http://www.bitwizard.nl/sig11/
We have been running a stand-alone memory test utility, "MEMTEST-86 v1.1", to perceive errors. We have run this for periods as long as twelve(12) hours without any errors being flagged.
At this point in time, any/all suggestions welcome. If more information is needed, let me know. In the meantime, we will keep trying to get a stable H/W & S/W combination.
TIA,
Dave Worden <dworden@us.teltech.com>
|  |