lkml.org 
[lkml]   [2005]   [Oct]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: 3Ware 9500S-12 RAID controller -- poor performance
    I have seen some systems on which IRQ load balancing can have a detrimental
    effect on some devices such as gigabit Ethernet etc.

    You could try disabling both the irqbalance userspace daemon (if that's part
    of your distribution), and in-kernel IRQ balancing, if enabled
    (CONFIG_IRQBALANCE).

    For your NIC, try enabling NAPI interrupt mitigation, if available. This
    will significantly reduce the interrupt load under high traffic volume.

    I guess there's another obvious question that I forgot: Do you have the
    3ware cache enabled or disabled? Are your ext3 filesystems mounted with the
    'noatime' option?

    Regards,
    Ian Morgan

    --
    -------------------------------------------------------------------
    Ian E. Morgan Vice President & C.O.O. Webcon, Inc.
    imorgan at webcon dot ca PGP: #2DA40D07 www.webcon.ca
    * Customized Linux network solutions for your business *
    -------------------------------------------------------------------

    On Tue, 4 Oct 2005, subbie subbie wrote:

    > Dear Ian,
    >
    > 'noapic' was a recommendation by 3Ware / AMCC tech
    > support. It did not help at all, as expected.
    > Unfortunately they did not have any other
    > recommendations.
    >
    > I've now removed 'noapic' and unfortunately nothing
    > has changed, really. See current stats below.
    >
    > # cat /proc/interrupts
    > CPU0 CPU1
    > 0: 64713875 355 IO-APIC-edge timer
    > 2: 0 0 XT-PIC cascade
    > 8: 3 1 IO-APIC-edge rtc
    > 14: 7 6 IO-APIC-edge ide0
    > 16: 176847225 191855106 IO-APIC-level eth0
    > 18: 499139 336893 IO-APIC-level libata
    > 48: 31491551 22761438 IO-APIC-level 3w-9xxx
    > NMI: 0 0
    > LOC: 64049632 64155206
    > ERR: 0
    > MIS: 0
    >
    > # uptime
    > 08:39:14 up 2 days, 23:54, 8 users, load average:
    > 0.16, 0.13, 0.07
    >
    > I have also tried playing with the parameters our
    > friend Ville has mentioned in his post, nothing has
    > come out of it.
    >
    > I'm willing to give any developer here access to my
    > production machine so that they can see the situation
    > first hand. Performance is just aweful.
    >
    > I'm planning to ditch RAID5 on this card and try JBOD
    > and in spreading files evenly across my 12 disks,
    > hopefully this would give some benifit.
    >
    > Something is very wrong with this card / driver /
    > firmware and or kernel combination, hopefully someone
    > can help out.
    >
    > Much appreciated
    >
    > --- Ian Morgan <imorgan@webcon.ca> wrote:
    >
    >> Why are you booting with 'noapic'.. in my experience
    >> that will seriously
    >> impact interrupt performance. Use the APIC if you've
    >> got it, which in this
    >> case you definitely do.
    >>
    >> Yes, having your gigabit NIC and RAID controller on
    >> the same IRQ (in PIC
    >> mode) could definitely me a source of trouble.
    >>
    >> In your web server testing, were you using an
    >> external traffic generator or
    >> an on-host process? If you try on-host (eliminating
    >> the network throughput
    >> and related interrupts) does performance improve?
    >>
    >> So two biggest suggestions:
    >>
    >> - Use the APIC. It is your friend.
    >>
    >> - It looks like the 3ware card and gigabit nic are
    >> on different busses, but
    >> the pirq lines are being routed to the same legacy
    >> interrupt in PIC mode. So
    >> APIC mode should avoid that problem. If the
    >> controller and nic are actually
    >> on the same bus, separate them.
    >>
    >>
    >> Regards,
    >> Ian Morgan
    >>
    >> --
    >>
    > -------------------------------------------------------------------
    >> Ian E. Morgan Vice President & C.O.O.
    >> Webcon, Inc.
    >> imorgan at webcon dot ca PGP: #2DA40D07
    >> www.webcon.ca
    >> * Customized Linux network solutions for your
    >> business *
    >>
    > -------------------------------------------------------------------
    >>
    >>
    >> On Thu, 29 Sep 2005, subbie subbie wrote:
    >>
    >>> Dear list,
    >>>
    >>> After almost two weeks of experimentation, google
    >>> searches and reading of posts, bug reports and
    >>> discussions I'm still far from an answer. I'm
    >> hoping
    >>> someone on this list could shed some light on the
    >>> subject.
    >>>
    >>> I'm using a 3Ware 9500S-12 card and am able to
    >> produce
    >>> up to 400MB/s sustained read from my 12-disk 4.1TB
    >>> RAID5 SATA array, 128MB cache onboard, ext3
    >> formatted.
    >>> All is well when performing a single read -- it
    >>> works nice and fast.
    >>>
    >>> The system is a web server, serving mid-size files
    >>> (50MB, each, on average). All hell breaks loose
    >> when
    >>> doing many concurrent reads, anywhere between 200
    >> to
    >>> 400 concurrent streams things simply grind to a
    >> halt
    >>> and the system transfers a maximum of 12-14MB/s.
    >>>
    >>> I'm in the process of clearing up the array (this
    >>> would take some time) and restructuring it to JBOD
    >>> mode in order to use each disk individually. I
    >> will
    >>> use a filesystem more suitable to streaming large
    >>> files, such as XFS. But this would take time and
    >> I
    >>> would very much appreciate the advice of people in
    >> the
    >>> know if this is going to help at all. It's hard
    >> for
    >>> me to make extreme experimentation (deleting,
    >>> formatting, reformatting) as this is a productio n
    >>> system with many files that I have no other place
    >> to
    >>> dump until they can be safely removed. Though I'm
    >>> working on dumping them slowly to other, remote,
    >>> machines.
    >>>
    >>> I'm running the latest kernel, 2.6.13.2 and the
    >> latest
    >>> 3Ware driver, taken from the 3ware.com web site
    >> which
    >>> upon insmod, updates the card's firmware to the
    >> latest
    >>> version as well.
    >>>
    >>> In my experiments, I've tried using larger
    >> readahead,
    >>> currently at 16k (this helps, higher values do not
    >>> seem to help much), using the deadline scheduler
    >> for
    >>> this device, booting the system with the 'noapic'
    >>> option and playing with a bunch of VM tunable
    >>> parameters which I'm not sure that I should really
    >> be
    >>> touching. At the moment only the readahead
    >>> modification is used as the other stuff simply
    >> didn't
    >>> help at all.
    >>>
    >>> With the stock kernel shipped with my
    >> distribution,
    >>> 2.6.8 and its old 3ware driver things were just as
    >>> worse but manifested themselves differently.
    >> The
    >>> system was visibly (top, vmstat...) spending most
    >> of
    >>> its time in io-wait and load average was extremely
    >>> high, in the area of 10 to 20. With the recent
    >>> kernel and driver mentioned above, the excessive
    >>> io-wait and load seems to have been resolved and
    >>> observed loadavg is between 1 and 4.
    >>>
    >>> I don't have much experience with systems that are
    >>> supposed to stream many files concurrently off a
    >>> hardware RAID of this configuration, but my gut
    >>> feeling is that something is very wrong and I
    >> should
    >>> be seeing a much higher read throughput.
    >>>
    >>> Trying to preempt people's questions I've tried to
    >>> include as much information as possible, a lot of
    >>> stuff is pasted below.
    >>>
    >>> I've just seen that the 3ware driver shares the
    >> same
    >>> IRQ with my ethernet card, which has got me a
    >> little
    >>> worried, should I be?
    >>>
    >>> System uptime, exactly 1 day:
    >>>
    >>> # cat /proc/interrupts
    >>> CPU0 CPU1
    >>> 0: 21619638 0 XT-PIC timer
    >>> 2: 0 0 XT-PIC cascade
    >>> 8: 4 0 XT-PIC rtc
    >>> 10: 268753224 0 XT-PIC
    >> 3w-9xxx,
    >>> eth0
    >>> 14: 11 0 XT-PIC ide0
    >>> 15: 337881 0 XT-PIC libata
    >>> NMI: 0 0
    >>> LOC: 21110402 21557685
    >>> ERR: 0
    >>> MIS: 0
    >>>
    >>> # free
    >>> total used free shared
    >>> buffers cached
    >>> Mem: 2075260 2024724 50536
    >> 0
    >>> 5184 1388408
    >>> -/+ buffers/cache: 631132 1444128
    >>> Swap: 3903784 0 3903784
    >>>
    >>> # vmstat -n 1 (output of the last few seconds):
    >>> procs -----------memory---------- ---swap--
    >>> -----io---- --system-- ----cpu----
    >>> 0 0 0 49932 4760 1392980 0 0 15636
    >>> 32 3169 3697 4 6 30 60
    >>> 0 0 0 50816 4752 1392376 0 0 5844
    >>> 0 3114 3929 3 5 91 1
    >>> 0 0 0 50924 4772 1391404 0 0 9360
    >>> 0 3187 4348 6 6 76 13
    >>> 0 2 0 50552 4780 1391532 0 0 24976
    >>> 44 4077 3906 3 7 65 25
    >>> 0 1 0 50444 4780 1392688 0 0 20192
    >>> 0 5048 3914 7 8 56 30
    >>> 0 1 0 50568 4756 1392508 0 0 21248
    >>> 0 4060 3603 4 6 48 41
    >>> 0 0 0 50704 4724 1392268 0 0 30004
    >>> 0 3834 3369 4 9 65 22
    >>> 0 3 0 50556 4728 1392468 0 0 3248
    >>> 1832 2974 4514 2 5 58 35
    >>> 0 3 0 50308 4724 1392200 0 0 1288
    >>> 336 1766 1886 1 3 50 47
    >>> 0 4 0 50308 4732 1391852 0 0 2300
    >>> 408 1919 2158 0 3 51 46
    >>> 0 4 0 50556 4736 1390692 0 0 1856
    >>> 532 1488 1846 3 1 50 46
    >>> 0 3 0 50680 4740 1390620 0 0 4016
    >>> 1296 1577 1682 2 2 50 47
    >>> 0 3 0 50432 4752 1391628 0 0 2180
    >>> 72 1730 1945 2 2 51 46
    >>> 2 2 0 49924 4772 1391540 0 0 44372
    >>> 564 3403 2847 4 5 50 42
    >>> 0 0 0 50684 4784 1391528 0 0 28640
    >>> 216 3804 3847 7 8 69 16
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2005-10-05 15:43    [W:0.045 / U:4.320 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site