[lkml]   [2009]   [Apr]   [2]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectI/O throughput problem in newer kernels
    While putting together a not so average machine for database research, I
    bumped into the following performance problem with the newer kernels (I
    tested, 2.6.29): the aggregate throughput drops drastically when
    more than 20 hard drives are involved in the operation. This problem is
    not happening on or 2.6.20 (did not test other kernels).

    Since I am not subscribed to the mailing list, I would appreciate you
    cc-ing me on any reply or discussion.

    1. Description of the machine
    8 Quad-Core AMD Opteron(tm) Processor 8346 HE
    Each processor has independent memory banks (16GB in each bank for 128GB
    Two PCI busses (connected in different places in the NUMA architecture)
    8 hard drives installed into the base system on SATA interfaces
    First hard drive dedicated to the OS
    7 Western Digital hard drives (90 MB/s max throughput)
    Nvidia SATA chipset
    4 Adaptec 5805 RAID cards installed in PCI-E 16X slots (all running at 8X
    The 4 cards live on two separate PCI busses
    6 IBM EXP3000 disk enclosures
    2 cards connect to 2 enclosures each, the other 2 to 1 enclosure
    8 Western Digital Velociraptor HD in each enclosure
    Max measured throughput 110-120 MB/s

    Total number of hard drives used the tests: 7+47=54 or subsets
    The Adaptec cards are configure to expose each disk individually to the
    OS. Any RAID configuration seems to limit the throughput at 300-350MB/s
    which is too low for the purpose of this system.

    2. Throughput tests
    I did two types of tests: using dd (spawning parallel dd jobs that lasted
    at least 10s) or using a multi-threaded program that simulates the
    intended usage for the system. Results using both are consistent so I will
    only report the results with the custom program. Both the dd test and the
    custom one do reads in large chunks (256K/request at least). All request
    in the custom program are made with "read" system call to page aligned
    memory (allocated with mmap to make sure). The kernel is doing a zero-copy
    to user space otherwise the speeds observed are not possible.

    Here is what I observed in terms of throughput:
    a. Speed/WD disk: 90 MB/s
    b. Speed/Velociraptor disk: 110 MB/s
    c. Speed of all WD disks in base system: 700MB/s
    d. Speed of disks in one enclosure: 750 MB/s
    e. Speed of disks connected to one Adaptec card: 1000 MB/s
    f. Speed of disks connected on a single PCI bus: 2000 MB/s

    The above numbers look good and are consistent on all kernels that I tried.

    THE PROBLEM: when the number of disks exceeds 20 the throughput plummets
    on newer kernels.

    g. SPEED OF ALL DISKS: 600 MB/s on newer kernels, 2700 MB/s on older kernels
    The throughput drops drastically the moment 20-25 hard drives are involved

    3. Tests I performed to ensure the number of hard drives is the culprit
    a. Took 1, 2, 3 and 4 disks from each enclosure to ensure uniform load on
    performance going up as expected until 20 drives reached than dropping

    b. Involved combinations of the regular WD drives and the Velociraptors.
    Had no major influence on the observation

    c. Involved combinations of enclosures
    No influence

    d. Used the hard drives in decreasing order of measured speed (as reported
    by hdparm)
    Only minor influence and still drastic drop at 20

    e. Changed the I/O scheduler used for the hard drives
    No influence

    4. Things that I do not think are wrong
    a. aacraid or scsi_nv drivers
    The problem depends only on the number of hard drives not the
    combination of the drives themselves

    b. Limitations on the buses
    The measured speeds of the subsystems indicate that no bottleneck on
    individual buses is reached. Even if this is the case, the throughput
    should level up not drop dramatically

    c. Failures in the system
    No errors reported in /var/log/messages or other logs related to I/O

    Of course, this begs the question WHAT IS WRONG?

    I would be more than happy to run any tests you suggest on my system to
    find the problem.


    Alin Dobra
    Assistant Professor
    Computer Information Science & Engineering Department
    University of Florida

     \ /
      Last update: 2009-04-02 17:29    [W:0.025 / U:8.840 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site