Messages in this thread | | | Subject | 2.4.27: md RAID1 oops on alpha revisited | From | Ingvar Hagelund <> | Date | 28 Sep 2004 01:44:24 +0200 |
| |
2.4.27: md RAID1 oops on alpha revisited
We have a Compaq Alphaserver DS10 466 MHz running Debian Woody with a self compiled 2.4.27 from kernel.org. We have not been able to make it run stable on md RAID1. It always crashes in less than an hour uptime, presumely while stressing the RAID code. Running on single disks, it's rock stable.
While debugging the problem and googling for answers, we came across very similar problems, posted to lkml by Norbert Preining 2004-03-27 (http://www.uwsg.iu.edu/hypermail/linux/kernel/0403.3/0837.html)
According to that thread, things should work ok with a newer gcc. It also seems like some version of the patch mentioned has gone into the kernel. We rebuilt the kernel using gcc-3.3.4. The result: The machine crashes maybe a little later, but always within an hour or so.
As the bootlog shows, we booted with degraded mirrors, and added the other parts of the mirrors after booting. (This to make sure the box doesn't oops till it's done booting.)
The machine locks up after the oops, so we had to run ksymoops after a reboot without raid, but with the same kernel and same modules loaded.
Below is ver_linux, cpuinfo, meminfo, some info from gcc, output from lsmod, the oops run though ksymoops, and the bootlog from dmesg. The oops was captured using a serial console.
Is it possible to make this thing run stable on md RAID1?
Ingvar
# sh /usr/src/linux-2.4.27/scripts/ver_linux If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes.
Linux nope 2.4.27-gcc334 #1 Fri Aug 27 12:14:37 CEST 2004 alpha unknown
Gnu C 2.95.4 Gnu make 3.79.1 util-linux 2.11n mount 2.11n modutils 2.4.26 e2fsprogs 1.27 PPP 2.4.1 Linux C Library 2.2.5 Dynamic linker (ldd) 2.2.5 Procps 2.0.7 Net-tools 1.60 Console-tools 0.2.3 Sh-utils 2.0.11 Modules Loaded ipt_LOG ipt_state ipt_TCPMSS iptable_mangle iptable_nat ip_conntrack iptable_filter ip_tables 8021q tulip crc32 rtc
# cat /proc/cpuinfo cpu : Alpha cpu model : EV6 cpu variation : 7 cpu revision : 0 cpu serial number : system type : Tsunami system variation : Webbrick system revision : 0 system serial number : cycle frequency [Hz] : 462962962 timer frequency [Hz] : 1024.00 page size [bytes] : 8192 phys. address bits : 44 max. addr. space # : 255 BogoMIPS : 921.84 kernel unaligned acc : 0 (pc=0,va=0) user unaligned acc : 0 (pc=0,va=0) platform string : COMPAQ AlphaServer DS10 466 MHz cpus detected : 1
# cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 523632640 46292992 477339648 0 4931584 22994944 Swap: 1077395456 0 1077395456 MemTotal: 511360 kB MemFree: 466152 kB MemShared: 0 kB Buffers: 4816 kB Cached: 22456 kB SwapCached: 0 kB Active: 15720 kB Inactive: 16824 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 511360 kB LowFree: 466152 kB SwapTotal: 1052144 kB SwapFree: 1052144 kB
# /usr/local/toolchain/bin/gcc-3.3 -dumpmachine alphaev6-unknown-linux-gnu # /usr/local/toolchain/bin/gcc-3.3 -dumpversion 3.3.4
# lsmod Module Size Used by Not tainted ipt_LOG 4697 1 (autoclean) ipt_state 880 2 (autoclean) ipt_TCPMSS 3032 1 (autoclean) iptable_mangle 2872 1 (autoclean) iptable_nat 23178 1 (autoclean) ip_conntrack 26297 0 (autoclean) [ipt_state iptable_nat] iptable_filter 2308 1 (autoclean) ip_tables 17408 8 [ipt_LOG ipt_state ipt_TCPMSS iptable_mangle iptable_nat iptable_filter] 8021q 19408 1 (autoclean) tulip 51048 2 crc32 3376 0 [tulip] rtc 6592 0 (autoclean)
ksymoops 2.4.5 on alpha 2.4.27-gcc334. Options used -V (specified) -k /var/log/ksymoops/20040928000118.ksyms (specified) -l /var/log/ksymoops/20040928000118.modules (specified) -o /lib/modules/2.4.27-gcc334/ (default) -m /boot/System.map-2.4.27-gcc334 (default)
Warning (compare_maps): mismatch on symbol ip_conntrack_destroyed , ip_conntrack says fffffffc00358680, /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o says fffffffc00352010. Ignoring /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o entry Warning (compare_maps): mismatch on symbol ip_conntrack_hash , ip_conntrack says fffffffc003586a0, /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o says fffffffc00352030. Ignoring /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o entry Warning (compare_maps): mismatch on symbol ip_conntrack_htable_size , ip_conntrack says fffffffc00358688, /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o says fffffffc00352018. Ignoring /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o entry Warning (compare_maps): mismatch on symbol ip_conntrack_lock , ip_conntrack says fffffffc00358678, /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o says fffffffc00352008. Ignoring /lib/modules/2.4.27-gcc334/kernel/net/ipv4/netfilter/ip_conntrack.o entry Warning (compare_maps): mismatch on symbol vlan_default_dev_flags , 8021q says fffffffc0033aaaa, /lib/modules/2.4.27-gcc334/kernel/net/8021q/8021q.o says fffffffc0033600a. Ignoring /lib/modules/2.4.27-gcc334/kernel/net/8021q/8021q.o entry Warning (compare_maps): mismatch on symbol vlan_group_lock , 8021q says fffffffc0033aab8, /lib/modules/2.4.27-gcc334/kernel/net/8021q/8021q.o says fffffffc00336000. Ignoring /lib/modules/2.4.27-gcc334/kernel/net/8021q/8021q.o entry Warning (compare_maps): mismatch on symbol vlan_name_type , 8021q says fffffffc0033aaa8, /lib/modules/2.4.27-gcc334/kernel/net/8021q/8021q.o says fffffffc00336008. Ignoring /lib/modules/2.4.27-gcc334/kernel/net/8021q/8021q.o entry Warning (compare_maps): mismatch on symbol tulip_debug , tulip says fffffffc003326bc, /lib/modules/2.4.27-gcc334/kernel/drivers/net/tulip/tulip.o says fffffffc0032600c. Ignoring /lib/modules/2.4.27-gcc334/kernel/drivers/net/tulip/tulip.o entry Warning (compare_maps): mismatch on symbol tulip_max_interrupt_work , tulip says fffffffc003326f8, /lib/modules/2.4.27-gcc334/kernel/drivers/net/tulip/tulip.o says fffffffc00326018. Ignoring /lib/modules/2.4.27-gcc334/kernel/drivers/net/tulip/tulip.o entry Warning (compare_maps): mismatch on symbol tulip_rx_copybreak , tulip says fffffffc003326fc, /lib/modules/2.4.27-gcc334/kernel/drivers/net/tulip/tulip.o says fffffffc0032601c. Ignoring /lib/modules/2.4.27-gcc334/kernel/drivers/net/tulip/tulip.o entry Unable to handle kernel paging request at virtual address 0000000000000240 swapper(0): Oops 1 pc = [<fffffc000050da04>] ra = [<fffffc000050da04>] ps = 0007 Not tainted Using defaults from ksymoops -t elf64-alpha -a alpha v0 = 0000000000000000 t0 = 0000000000000000 t1 = fffffc000050dc10 t2 = 0000000000000000 t3 = fffffc001fdd6000 t4 = fffffc000064cdf8 t5 = fffffffffffffc18 t6 = ffffffffffffffff t7 = fffffc0000600000 s0 = 0000000000000000 s1 = 0000000000000004 s2 = fffffc001ff86800 s3 = fffffc001ff868b8 s4 = 0000000000000004 s5 = fffffc00006eef40 s6 = 0000000000000000 a0 = fffffc001fdd60c0 a1 = 0000000000000000 a2 = fffffc0000603ec8 a3 = fffffc0000b70158 a4 = 000000011ffffa20 a5 = 0000000000000000 t8 = 000000012000e8ad t9 = 00000200000ccf84 t10= 000000000000000b t11= 0000000000000400 pv = fffffc000050d6b0 at = fffffc0000605000 gp = fffffc00006e1a00 sp = fffffc0000603df8 Trace:fffffc000050d6dc fffffc0000317ca4 fffffc0000318640 fffffc000031ff70 fffffc0000318c84 fffffc0000313308 fffffc00003148c0 fffffc0000320200 fffffc00003148a0 fffffc00003100a0 fffffc000031001c Code: 23bd4018 c3ffffc8 2ffe0000 47ff041f 2ffe0000 d3400049 <b0090240> c3ffffbd
>>RA; fffffc000050da04 <isp1020_intr_handler+304/420>
>>PC; fffffc000050da04 <isp1020_intr_handler+304/420> <=====
Trace; fffffc000050d6dc <do_isp1020_intr_handler+2c/50> Trace; fffffc0000317ca4 <handle_IRQ_event+a4/120> Trace; fffffc0000318640 <handle_irq+d0/190> Trace; fffffc000031ff70 <dp264_srm_device_interrupt+30/50> Trace; fffffc0000318c84 <do_entInt+f4/150> Trace; fffffc0000313308 <ret_from_sys_call+0/10> Trace; fffffc00003148c0 <cpu_idle+70/80> Trace; fffffc0000320200 <do_check_pgt_cache+0/140> Trace; fffffc00003148a0 <cpu_idle+50/80> Trace; fffffc00003100a0 <rest_init+40/60> Trace; fffffc000031001c <_stext+1c/20>
Code; fffffc000050d9ec <isp1020_intr_handler+2ec/420> 0000000000000000 <_PC>: Code; fffffc000050d9ec <isp1020_intr_handler+2ec/420> 0: 18 40 bd 23 lda gp,16408(gp) Code; fffffc000050d9f0 <isp1020_intr_handler+2f0/420> 4: c8 ff ff c3 br ffffffffffffff28 <_PC+0xffffffffffffff28> fffffc000050d914 <isp1020_intr_handler+214/420> Code; fffffc000050d9f4 <isp1020_intr_handler+2f4/420> 8: 00 00 fe 2f unop Code; fffffc000050d9f8 <isp1020_intr_handler+2f8/420> c: 1f 04 ff 47 nop Code; fffffc000050d9fc <isp1020_intr_handler+2fc/420> 10: 00 00 fe 2f unop Code; fffffc000050da00 <isp1020_intr_handler+300/420> 14: 49 00 40 d3 bsr ra,13c <_PC+0x13c> fffffc000050db28 <isp1020_return_status+8/100> Code; fffffc000050da04 <isp1020_intr_handler+304/420> <===== 18: 40 02 09 b0 stl v0,576(s0) <===== Code; fffffc000050da08 <isp1020_intr_handler+308/420> 1c: bd ff ff c3 br ffffffffffffff14 <_PC+0xffffffffffffff14> fffffc000050d900 <isp1020_intr_handler+200/420>
Kernel panic: Aiee, killing interrupt handler!
10 warnings issued. Results may not be reliable.
# dmesg Linux version 2.4.27-gcc334 (foo@nope) (gcc version 3.3.4) #1 Fri Aug 27 12:14:37 CEST 2004 Booting on Tsunami variation Webbrick using machine vector Webbrick from SRM Major Options: LEGACY_START MAGIC_SYSRQ Command line: ro root=/dev/md1 md=1,/dev/sda1,missing md=2,/dev/sda2,missing md=3,/dev/sda3,missing md=4,/dev/sda4,missing console=ttyS0,57600 memcluster 0, usage 1, start 0, end 256 memcluster 1, usage 0, start 256, end 65479 memcluster 2, usage 1, start 65479, end 65536 freeing pages 256:384 freeing pages 932:65479 reserving pages 932:933 On node 0 totalpages: 65479 zone(0): 65479 pages. zone(1): 0 pages. zone(2): 0 pages. Kernel command line: ro root=/dev/md1 md=1,/dev/sda1,missing md=2,/dev/sda2,missing md=3,/dev/sda3,missing md=4,/dev/sda4,missing console=ttyS0,57600 md: Will configure md1 (super-block) from /dev/sda1,missing, below. md: Will configure md2 (super-block) from /dev/sda2,missing, below. md: Will configure md3 (super-block) from /dev/sda3,missing, below. md: Will configure md4 (super-block) from /dev/sda4,missing, below. Using epoch = 2000 Console: colour VGA+ 80x25 Calibrating delay loop... 921.84 BogoMIPS Memory: 511208k/523832k available (2803k kernel code, 10576k reserved, 889k data, 152k init) Dentry cache hash table entries: 65536 (order: 7, 1048576 bytes) Inode cache hash table entries: 32768 (order: 6, 524288 bytes) Mount cache hash table entries: 512 (order: 0, 8192 bytes) Buffer cache hash table entries: 32768 (order: 5, 262144 bytes) Page-cache hash table entries: 65536 (order: 6, 524288 bytes) POSIX conformance testing by UNIFIX Linux NET4.0 for Linux 2.4 Based upon Swansea University Computer Society NET3.039 Initializing RT netlink socket Starting kswapd VFS: Disk quotas vdquot_6.5.1 Journalled Block Device driver loaded SGI XFS with no debug enabled pty: 256 Unix98 ptys configured Serial driver version 5.05c (2001-07-08) with MANY_PORTS MULTIPORT SHARE_IRQ SERIAL_PCI enabled ttyS00 at 0x03f8 (irq = 4) is a 16550A ttyS01 at 0x02f8 (irq = 3) is a 16550A RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx ALI15X3: IDE controller at PCI slot 00:0d.0 ALI15X3: chipset revision 193 ALI15X3: not 100% native mode: will probe irqs later ide0: BM-DMA at 0x8800-0x8807, BIOS settings: hda:pio, hdb:pio ide1: BM-DMA at 0x8808-0x880f, BIOS settings: hdc:pio, hdd:pio hdc: COMPAQ CDR-8435, ATAPI CD/DVD-ROM drive ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx ide1 at 0x170-0x177,0x376 on irq 15 SCSI subsystem driver Revision: 1.00 qlogicisp : new isp1020 revision ID (5) scsi0 : QLogic ISP1020 SCSI on PCI bus 00 device 88 irq 47 MEM base 0xfffffd000a0c1000 Vendor: SEAGATE Model: ST373307LW Rev: 0007 Type: Direct-Access ANSI SCSI revision: 03 Vendor: SEAGATE Model: ST373307LW Rev: 0007 Type: Direct-Access ANSI SCSI revision: 03 DC390: 0 adapters found Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0 SCSI device sda: 143374744 512-byte hdwr sectors (73408 MB) Partition check: sda: sda1 sda2 sda3 sda4 SCSI device sdb: 143374744 512-byte hdwr sectors (73408 MB) sdb: sdb1 sdb2 sdb3 sdb4 md: linear personality registered as nr 1 md: raid0 personality registered as nr 2 md: raid1 personality registered as nr 3 md: raid5 personality registered as nr 4 raid5: measuring checksumming speed 8regs : 1843.200 MB/sec 32regs : 2121.728 MB/sec alpha : 2039.808 MB/sec alpha prefetch: 1925.120 MB/sec raid5: using function: alpha prefetch (1925.120 MB/sec) md: multipath personality registered as nr 7 md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27 md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE. md: Unknown device name: missing md: Loading md1: /dev/sda1 [events: 00000054] md: bind<sda1,1> md: sda1's event counter: 00000054 md: RAID level 1 does not need chunksize! Continuing anyway. md1: max total readahead window set to 248k md1: 1 data-disks, max readahead per data-disk: 248k raid1: device sda1 operational as mirror 0 raid1: md1, not all disks are operational -- trying to recover array raid1: raid set md1 active with 1 out of 2 mirrors md: updating md1 RAID superblock on device md: sda1 [events: 00000055]<6>(write) sda1's sb offset: 2104448 md: recovery thread got woken up ... md1: no spare disk to reconstruct array! -- continuing in degraded mode md: recovery thread finished ... md: Unknown device name: missing md: Loading md2: /dev/sda2 [events: 0000003b] md: bind<sda2,1> md: sda2's event counter: 0000003b md: RAID level 1 does not need chunksize! Continuing anyway. md2: max total readahead window set to 248k md2: 1 data-disks, max readahead per data-disk: 248k raid1: device sda2 operational as mirror 0 raid1: md2, not all disks are operational -- trying to recover array raid1: raid set md2 active with 1 out of 2 mirrors md: updating md2 RAID superblock on device md: sda2 [events: 0000003c]<6>(write) sda2's sb offset: 4200896 md: recovery thread got woken up ... md2: no spare disk to reconstruct array! -- continuing in degraded mode md1: no spare disk to reconstruct array! -- continuing in degraded mode md: recovery thread finished ... md: Unknown device name: missing md: Loading md3: /dev/sda3 [events: 00000039] md: bind<sda3,1> md: sda3's event counter: 00000039 md: RAID level 1 does not need chunksize! Continuing anyway. md3: max total readahead window set to 248k md3: 1 data-disks, max readahead per data-disk: 248k raid1: device sda3 operational as mirror 1 raid1: md3, not all disks are operational -- trying to recover array raid1: raid set md3 active with 1 out of 2 mirrors md: updating md3 RAID superblock on device md: sda3 [events: 0000003a]<6>(write) sda3's sb offset: 4200896 md: recovery thread got woken up ... md3: no spare disk to reconstruct array! -- continuing in degraded mode md2: no spare disk to reconstruct array! -- continuing in degraded mode md1: no spare disk to reconstruct array! -- continuing in degraded mode md: recovery thread finished ... md: Unknown device name: missing md: Loading md4: /dev/sda4 [events: 0000003a] md: bind<sda4,1> md: sda4's event counter: 0000003a md: RAID level 1 does not need chunksize! Continuing anyway. md4: max total readahead window set to 248k md4: 1 data-disks, max readahead per data-disk: 248k raid1: device sda4 operational as mirror 1 raid1: md4, not all disks are operational -- trying to recover array raid1: raid set md4 active with 1 out of 2 mirrors md: updating md4 RAID superblock on device md: sda4 [events: 0000003b]<6>(write) sda4's sb offset: 1052160 md: recovery thread got woken up ... md4: no spare disk to reconstruct array! -- continuing in degraded mode md3: no spare disk to reconstruct array! -- continuing in degraded mode md2: no spare disk to reconstruct array! -- continuing in degraded mode md1: no spare disk to reconstruct array! -- continuing in degraded mode md: recovery thread finished ... Initializing Cryptographic API NET4: Linux TCP/IP 1.0 for NET4.0 IP Protocols: ICMP, UDP, TCP, IGMP IP: routing cache hash table of 8192 buckets, 64Kbytes TCP: Hash tables configured (established 65536 bind 65536) Linux IP multicast router 0.06 plus PIM-SM NET4: Unix domain sockets 1.0/SMP for Linux NET4.0. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Freeing unused kernel memory: 152k freed Adding Swap: 1052144k swap-space (priority -1) EXT3 FS 2.4-0.9.19, 19 August 2002 on md(9,1), internal journal rtc: SRM (post-2000) epoch (2000) detected Real Time Clock Driver v1.10f Linux Tulip driver version 0.9.15-pre12 (Aug 9, 2002) tulip0: EEPROM default media type Autosense. tulip0: Index #0 - Media 10baseT (#0) described by a 21142 Serial PHY (2) block. tulip0: Index #1 - Media 10baseT-FDX (#4) described by a 21142 Serial PHY (2) block. tulip0: Index #2 - Media 100baseTx (#3) described by a 21143 SYM PHY (4) block. tulip0: Index #3 - Media 100baseTx-FDX (#5) described by a 21143 SYM PHY (4) block. eth0: Digital DS21143 Tulip rev 65 at 0x8400, 08:00:2B:86:1C:54, IRQ 29. tulip1: EEPROM default media type Autosense. tulip1: Index #0 - Media 10baseT (#0) described by a 21142 Serial PHY (2) block. tulip1: Index #1 - Media 10baseT-FDX (#4) described by a 21142 Serial PHY (2) block. tulip1: Index #2 - Media 100baseTx (#3) described by a 21143 SYM PHY (4) block. tulip1: Index #3 - Media 100baseTx-FDX (#5) described by a 21143 SYM PHY (4) block. eth1: Digital DS21143 Tulip rev 65 at 0x8480, 08:00:2B:86:1C:55, IRQ 30. kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on md(9,2), internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on md(9,3), internal journal EXT3-fs: mounted filesystem with ordered data mode. 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com> All bugs added by David S. Miller <davem@redhat.com> eth0.205: add 01:00:5e:00:00:01 mcast address to master interface ip_tables: (C) 2000-2002 Netfilter core team ip_conntrack version 2.1 (2046 buckets, 16368 max) - 416 bytes per conntrack
-- Many that live deserve death. And some that die deserve life. Can you give it to them? Then do not be too eager to deal out death in judgement. For even the very wise cannot see all ends. Gandalf - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |