Messages in this thread | | | Date | Tue, 10 Mar 2009 11:16:32 -0400 | From | "Alan D. Brunelle" <> | Subject | PROBLEM: kernel BUG at mm/slab.c:3002! |
| |
Running blktrace & I/O loads cause a kernel BUG at mm/slab.c:3002!.
I'm running some moderate I/O loads (using 12 devices behind a Smart Array on a 16-way x86_64 box, I'm doing asynchronous direct sequential reads to each disk in parallel), and whilst attempting to get blktrace data I routinely run into this.
I first started seeing this in 2.6.29-rc6, so I bumped to 2.6.29-rc7 and made a couple of successful runs, then ran into it again. (I've attached the script I was using, but I'm not sure it's very helpful...)
The environment the system under test is in is rather difficult to bisect in, but if need be, I can certainly go through the (painful) motions to do so...
I only ran this a couple of times on 2.6.27.7-4 (SLESS 11 b6 kernel), and both times it worked there - not sure how far back the problem occurs...
I'm open to any SLAB debug tracing options that may help with this...
Alan D.Brunelle Hewlett-Packard If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes.
Linux seatpost 2.6.29-rc7 #3 SMP Tue Mar 10 10:15:07 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Gnu C 4.3 Gnu make 3.81 binutils 2.19 util-linux 2.14.1 mount support module-init-tools 3.4 e2fsprogs 1.41.1 reiserfsprogs 3.6.19 xfsprogs 2.10.1 quota-tools 3.16. PPP 2.4.5 Linux C Library 2.9 Dynamic linker (ldd) 2.9 Procps 3.2.7 Net-tools 1.60 Kbd 1.14.1 oprofile 0.9.4 Sh-utils 6.12 udev 128 wireless-tools 30 Modules Loaded fuse ext2 loop dm_mod sd_mod crc_t10dif bnx2 qla2xxx sg scsi_transport_fc rtc_cmos ipmi_si shpchp sr_mod rtc_core ipmi_msghandler pcspkr rtc_lib serio_raw hpwdt pci_hotplug cdrom button scsi_tgt container hpilo usbhid hid ehci_hcd uhci_hcd ohci_hcd usbcore edd ext3 mbcache jbd fan ide_pci_generic amd74xx ide_core pata_amd thermal processor thermal_sys hwmon cciss ata_generic libata scsi_mod #! /usr/bin/env python
import os, sys, tempfile, time import parse_aiod
dsfs = 'sdy sdz sdaa sdab sdac sdad sdae sdaf sdag sdah sdai sdaj'.split()
def run_aiod(bs, dsfs, rnd): def gen_aiod_cmd(bs, dsfs): aiod_cmd= '/usr/local/bin/aiod' opts = '-A 256 -D -I -i 10000000 -l -v' xopts = '-R -b %dk -T 30' % bs cmd = '%s %s %s' % (aiod_cmd, opts, xopts) if rnd: cmd = '%s -z' % cmd for path in ['/dev/%s' % dsf for dsf in dsfs]: cmd = '%s %s' % (cmd, path) return cmd
(fd, fn) = tempfile.mkstemp() fo = os.fdopen(fd, 'w')
pid = os.fork() if pid == 0: os.dup2(fo.fileno(), sys.stdout.fileno()) os.dup2(fo.fileno(), sys.stderr.fileno()) fo.close()
cmd = gen_aiod_cmd(bs, dsfs).split(None) os.execvp(cmd[0], cmd) sys.exit(1) else: fo.close() os.waitpid(pid, 0)
pa = parse_aiod.parse_aiod(fn) mbps = pa.get_mb_per_sec() del pa
os.unlink(fn) return mbps
if __name__ == '__main__': def start_bt(dir, dsfs): if os.path.exists(dir): os.system('/bin/rm -rf ' + dir) os.mkdir(dir) cmd = '/usr/local/bin/blktrace -D %s -b 1024 -n 6' % dir for dsf in dsfs: cmd = '%s /dev/%s' % (cmd, dsf)
pid = os.fork() if pid > 0: time.sleep(30) return pid
fo = open('%s/bt.out' % dir, 'w') os.dup2(fo.fileno(), sys.stdout.fileno()) os.dup2(fo.fileno(), sys.stderr.fileno()) fo.close()
cmd = cmd.split(None) os.execvp(cmd[0], cmd) sys.exit(1)
def stop_bt(pid): os.kill(pid, 1) os.waitpid(pid, 0)
#bss = [ 4, 8, 16, 32, 64, 128, 256 ] bss = [ 32, 64, 128 ]
print '%3s:' % 'run', for bs in bss: print '%6d ' % bs, print '\n----', for bs in bss: print '%6s ' % '------', print '' for run in range(0, 3): print '%3d:' % run, sys.stdout.flush() for bs in bss: pid = start_bt('%03dk_%03d' % (bs, run), dsfs) print '%6.2f ' % run_aiod(bs, dsfs, False), sys.stdout.flush() stop_bt(pid) print ''
sys.exit(0)
| |