lkml.org 
[lkml]   [2006]   [Mar]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    SubjectOOM kiler/load problems with RAID/LVM and AoE
    Date
    Right off: This e-mail is going to be long.  I'm first going to explain how I 
    first discovered this bug, then explain how I set up a "smaller scale" test
    case. And I apologize for the wandering nature of this message as I am
    trying to debug things and find the actual root cause of my problem. I've
    read through the entire LKML FAQ, so I hope all that follows is proper form.

    Initial setup:
    I am using two Coraid Etherdrive units each with 15 400GB drives. Each unit
    is configured internally as a RAID5 device, which presents a 5.1TB drive to
    the OS. I wanted to mirror these units via RAID1, but discovered the 2TB
    RAID component limit.

    So, to get around this, here is what I did:
    Put each unit into it's own LVM volume group (vg-shelf0 and vg-shelf1). Next,
    I split each VG up into 5 1TB logical volumes (p1 through p5). Then, I
    mirrored each 1TB LV against its corresponding LV in the other VG (e.g.
    vg-shelf0/p1 and vg-shelf1/p1). I then took those 5 RAID1 devices and
    assembled them into a RAID-Linear device to obtain a 5.1TB drive. All that
    worked fine.

    However, when I try to run stress tests against the system (using spew or
    bonnie++), after a while, the load will start increasing, and eventually the
    system will just hang. I've done this to make sure I don't run out of memory
    (because oom-killer was firing off):

    echo 5 > /proc/sys/vm/dirty_ratio
    echo 5 > /proc/sys/vm/dirty_background_ratio
    echo 2 > /proc/sys/vm/overcommit_memory
    echo 0 > /proc/sys/vm/overcommit_ratio

    That didn't seem to have any positive effect.

    No, to take out two variables. I created a similar setup with two internal
    400GB SATA hard drives (same brand, incidentally, as are in the Etherdrives),
    so we got rid of AoE and the network drivers.

    Here's what I did to test this process on the internal drives:

    Prep the drives to be added to a VG
    pvcreate /dev/sdc
    pvcreate /dev/sdd

    First, put each shelf in its own volume group
    vgcreate /dev/vg-test0 /dev/sdc
    vgcreate /dev/vg-test1 /dev/sdd
    vgchange -ay /dev/vg-test0
    vgchange -ay /dev/vg-test1

    We now split each VG into 4 100GB chunks
    Total PE = 95387 or 23846/23846/23846/23849

    lvcreate -l 23846 -n p1 vg-test0
    lvcreate -l 23846 -n p2 vg-test0
    lvcreate -l 23846 -n p3 vg-test0
    lvcreate -l 23849 -n p4 vg-test0

    lvcreate -l 23846 -n p1 vg-test1
    lvcreate -l 23846 -n p2 vg-test1
    lvcreate -l 23846 -n p3 vg-test1
    lvcreate -l 23849 -n p4 vg-test1

    Now, create 4 100GB RAID1 md devices:

    mdadm -C /dev/md7 --auto -l raid1 -n 2 /dev/vg-test0/p1 /dev/vg-test1/p1
    mdadm -C /dev/md8 --auto -l raid1 -n 2 /dev/vg-test0/p2 /dev/vg-test1/p2
    mdadm -C /dev/md9 --auto -l raid1 -n 2 /dev/vg-test0/p3 /dev/vg-test1/p3
    mdadm -C /dev/md10 --auto -l raid1 -n 2 /dev/vg-test0/p4 /dev/vg-test1/p4

    Now, we create a 400GB Linear device that is spread across the five RAID1
    devices.
    mdadm -C /dev/md11 --auto -l linear -n 4 /dev/md7 /dev/md8 /dev/md9 /dev/md10

    Now, running this command:

    bonnie++ -d /xtesting/bonnie -s 50g -u joshua:joshua

    is where I have problems. It does the "putc()" section fine, but when it
    starts "Writing intelligently," load goes through the roof, and the system
    eventually hangs. It exhibits this behavior on the Etherdrive setup above
    but *not* in the internal hard drive setup immediately above.

    I assume the LVM/RAID code is pretty well tested. Any hints as to where I
    might start debugging? syslog does not report anything useful (even
    on-screen syslog). The AoE driver could be at fault, but seeing as I copied
    95GB of files yesterday, I'm wondering if it has something to do with the
    network drivers. These are Broadcoms using the tg3 driver...I have heard of
    problems with these, correct? I put in a Intel Pro/1000 Dual NIC network
    card and connected to the Etherdrives using this interface (I'm using Cat6
    crossover cables, one Etherdrive per port). I re-ran the tests and had the
    same results.

    RAID or LVM problem? AoE drivers? Network driver badness (for both of them)?

    I shall attempt to include all useful info:

    Distribution: Mandriva 2006
    Kernel: Linux community.dist-ed.uaf.edu 2.6.12-14mdksmp #1 SMP Tue Dec 20
    13:45:20 MST 2005 i686 AMD Opteron(tm) Processor 246 unknown GNU/Linux
    AoE drivers: aoe module version 23
    AoE units are running on SS20060302 firmware
    Note: AMD Opterons running in 32bit mode
    Motherboard: Tyan S2882 Dual Opteron
    RAM: 8GB
    CPU's:
    processor : 0
    vendor_id : AuthenticAMD
    cpu family : 15
    model : 5
    model name : AMD Opteron(tm) Processor 246
    stepping : 10
    cpu MHz : 1992.068
    cache size : 1024 KB

    processor : 1
    vendor_id : AuthenticAMD
    cpu family : 15
    model : 5
    model name : AMD Opteron(tm) Processor 246
    stepping : 10
    cpu MHz : 1992.068
    cache size : 1024 KB

    GCC 4.0.1 was used to compile (I think), binutils-2.16.91.0.2-3mdk (again, I
    think). Not 100% sure as this is an RPM installed kernel, so I don't know
    what versions were used to compile it.

    [root@community /xtesting/bonnie]# lspcidrake
    unknown : Advanced Micro Devices|AMD-8111 PCI [BRIDGE_PCI]
    amd76xrom : Advanced Micro Devices|AMD-8111 LPC [BRIDGE_ISA]
    amd74xx : Advanced Micro Devices|AMD-8111 IDE [STORAGE_IDE]
    i2c-amd8111 : Advanced Micro Devices|AMD-8111 SMBus 2.0 [SERIAL_SMBUS]
    hw_random : Advanced Micro Devices|AMD-8111 ACPI [BRIDGE_OTHER]
    unknown : Advanced Micro Devices|AMD-8131 PCI-X Bridge [BRIDGE_PCI]
    unknown : Advanced Micro Devices|AMD-8131 PCI-X Bridge [BRIDGE_PCI]
    unknown : Advanced Micro Devices|AMD-8131 PCI-X APIC [SYSTEM_PIC]
    unknown : Advanced Micro Devices|AMD-8131 PCI-X APIC [SYSTEM_PIC]
    unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] HyperTransport
    Technology Configuration [BRIDGE_HOST]
    unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] HyperTransport
    Technology Configuration [BRIDGE_HOST]
    unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] Address Map
    [BRIDGE_HOST]
    unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] Address Map
    [BRIDGE_HOST]
    unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] DRAM Controller
    [BRIDGE_HOST]
    unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] DRAM Controller
    [BRIDGE_HOST]
    amd64-agp : Advanced Micro Devices|K8 [Athlon64/Opteron] Miscellaneous
    Control [BRIDGE_HOST]
    amd64-agp : Advanced Micro Devices|K8 [Athlon64/Opteron] Miscellaneous
    Control [BRIDGE_HOST]
    usb-ohci : Advanced Micro Devices|AMD-8111 USB [SERIAL_USB]
    usb-ohci : Advanced Micro Devices|AMD-8111 USB [SERIAL_USB]
    sata_sil : Silicon Image|Sil3114A Serial ATA [STORAGE_OTHER]
    Card:ATI Rage XL: ATI|Rage XL [DISPLAY_VGA]
    tg3 : Broadcom Corp.|BCM5704 CIOB-E 1000BaseTX [NETWORK_ETHERNET]
    tg3 : Broadcom Corp.|BCM5704 CIOB-E 1000BaseTX [NETWORK_ETHERNET]
    e1000 : Intel Corp.|82544EI Gigabit Ethernet Controller
    [NETWORK_ETHERNET]
    hub : Linux 2.6.12-14mdksmp ohci_hcd|OHCI Host Controller [Hub|
    Unused]
    hub : Linux 2.6.12-14mdksmp ohci_hcd|OHCI Host Controller [Hub|
    Unused]

    j----- k-----

    --
    Joshua Kugler PGP Key: http://pgp.mit.edu/
    CDE System Administrator ID 0xDB26D7CE
    http://distance.uaf.edu/
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2006-03-14 02:05    [W:0.033 / U:3.028 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site