lkml.org 
[lkml]   [2004]   [Jun]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Subject2.4.26 SMP lockup problem
Date

Hello All.

During an interesting round of kernel updates, I found a very interesting
problem. I have several "hundred" nodes in a cluster that I am currently
updating from kernel 2.4.21 to 2.4.26. These nodes are all running RedHat
7.3 (old, I know, but this is the OS that are software currently works on).
During this round of updates, I have updated about 150 PIII 800 MHz nodes,
all of which are currently being used and work just fine (1 GB Ram, e100
ethernet driver, IDE drives, fairly generic). Also, I have a few PIII 1260
nodes (Tyan Motherboard, 2 GB Ram, e100 ethernet driver, again, fairly
generic) that have also been updated and run fine. I have even started
testing fairly new P4 3060 IBM blades. They also seem to work just fine.

Now to the problem. I have "several hundred" Tyan Thunder Motherboards (older
AMD 760MP chipset). I have rebooted ~ 200 of these nodes with the new 2.4.26
kernel and about half of these nodes have suffered a hard lockup during
bootup. The lockup is hard enough that I cannot even isuse sys request keys
over serial or at the local keyboard to cause them to reboot or output a
trace. These nodes have 2 GB of ram, dual 3Com 100 Mb NICS, and IDE drives.
Again, fairly generic for a cluster. I had a vanilal + trond patched 2.4.21
kernel running on these boxes just fine. (The new 2.4.26 kernel also has the
trond patches for 2.4.26). Has anyone seen this happen to them?

Here is some info on the kernel config for the 2.4.26 kernel:

#
# Automatically generated by make menuconfig: don't edit
#
CONFIG_X86=y
# CONFIG_SBUS is not set
CONFIG_UID16=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODVERSIONS=y
CONFIG_KMOD=y

#
# Processor type and features
#
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
CONFIG_MPENTIUMIII=y
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MELAN is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_X86_HAS_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_PGE=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_F00F_WORKS_OK=y
CONFIG_X86_MCE=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_MICROCODE=y
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_EDD is not set
# CONFIG_NOHIGHMEM is not set
# CONFIG_HIGHMEM4G is not set
CONFIG_HIGHMEM64G=y
CONFIG_HIGHMEM=y
CONFIG_X86_PAE=y
CONFIG_HIGHIO=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
# CONFIG_X86_NUMA is not set
# CONFIG_X86_TSC_DISABLE is not set
CONFIG_X86_TSC=y
CONFIG_HAVE_DEC_LOCK=y

#
# General setup
#
CONFIG_NET=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_ISA=y
CONFIG_PCI_NAMES=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
CONFIG_HOTPLUG=y
---- Rest cut -------

I have the noapic option passed on the lilo boot prompt line, otherwise we get
the APIC error after about a month or two in service.

We tried to make the kernel somewhat generic because we want this kernel to
boot on the largest hardware base possible. Is there something obvious that
I have missed (I have used these options on the 2.4.21 kernel that we used on
all of the nodes with the exception of the 64 GB memory.

Any help would be appreciated. Any dumps that need to be made (or try to
make), great as I have about 200 nodes right now that are candidates for
testing.

Please contact me at email listed below as I am not on the list.


Email: norman.r.weathers@conocophillips.com


Thanks in advance.

--

Norman Weathers
SIP Linux Cluster
TCE UNIX
ConocoPhillips
Houston, TX

Office: LO2003
Phone: ETN 639-2727
or (281) 293-2727
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 14:03    [W:0.055 / U:0.460 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site