lkml.org 
[lkml]   [2011]   [Feb]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?
    Dne 6.2.2011 14:46, Justin Piszcz napsal(a):
    >
    >
    > On Sun, 6 Feb 2011, Justin Piszcz wrote:
    >
    >>
    >>
    >> On Sat, 5 Feb 2011, Stan Hoeppner wrote:
    >>
    >>> Justin Piszcz put forth on 2/5/2011 7:08 PM:
    >>>
    >
    >
    > Hi,

    Hi, just few comments for maximal throughput..

    > 1. Defaults below:
    > sysctl -w net.core.wmem_max=131071
    > sysctl -w net.core.rmem_max=131071
    > sysctl -w net.core.wmem_default=118784
    > sysctl -w net.core.rmem_default=118784
    > sysctl -w net.core.optmem_max=20480
    > sysctl -w net.ipv4.igmp_max_memberships=20
    > sysctl -w net.ipv4.tcp_mem="379104 505472 758208"
    > sysctl -w net.ipv4.tcp_wmem="4096 16384 4194304"
    > sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"
    > sysctl -w net.ipv4.udp_mem="379104 505472 758208"
    > sysctl -w net.ipv4.udp_rmem_min=4096
    > sysctl -w net.ipv4.udp_wmem_min=4096
    > sysctl -w net.core.netdev_max_backlog=1024

    sysctl net.ipv4.tcp_timestamps=0

    > 2. Optimized settings, for > 800MiB/:
    >
    > # for 3ware raid, use 16384 readahead, > 16384 readahead, no improvement
    > blockdev --setra 16384 /dev/sda

    elevator=deadline

    > # not sure if this helps much
    > ethtool -K eth0 lro on

    Maybe try to _disable_ NIC offloads functions, sometimes its contra
    productive (with enough CPU power, but I doubt on 2 socket box) + check
    irqbalance..

    If you have connection just between machines try the biggest possible MTU.

    > # seems to get performance > 600-700MiB/s faster
    > sysctl -w net.core.wmem_max=4194304
    > sysctl -w net.core.rmem_max=4194304
    > sysctl -w net.core.wmem_default=4194304
    > sysctl -w net.core.rmem_default=4194304
    > sysctl -w net.core.optmem_max=20480
    > sysctl -w net.ipv4.igmp_max_memberships=20
    > sysctl -w net.ipv4.tcp_mem="4194304 4194304 4194304"
    > sysctl -w net.ipv4.tcp_wmem="4194304 4194304 4194304"
    > sysctl -w net.ipv4.tcp_rmem="4194304 4194304 4194304"
    > sysctl -w net.ipv4.udp_mem="4194304 4194304 4194304"
    > sysctl -w net.ipv4.udp_rmem_min=4096
    > sysctl -w net.ipv4.udp_wmem_min=4096
    > sysctl -w net.core.netdev_max_backlog=1048576
    >
    > # the main option that makes all of the difference, the golden option
    > # is the rszie and wsize of 1megabyte below:
    > 10.0.1.4:/r1 /nfs/box2/r1 nfs
    > tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=1048576,wsize=1048576 0 0
    >
    > CPU utilization:
    > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    > 2069 root 20 0 18640 1304 688 R 91 0.0 0:15.50 cp
    > 703 root 20 0 0 0 0 S 25 0.0 2:46.95 kswapd0
    >
    > With a single copy I get roughly 700-800MiB/s:
    >
    > Device eth0 [10.0.1.3] (1/1):
    > ================================================================================
    >
    > Incoming:
    > ###################### #################### ####
    > ###################### #################### ####
    > ###################### #################### ####
    > ###################### #################### ####
    > ###################### #################### ####
    > ###################### #################### #### Curr: 808.71 MByte/s
    > ###################### #################### #### Avg: 706.11 MByte/s
    > ###################### #################### #### Min: 0.00 MByte/s
    > ###################### #################### #### Max: 860.17 MByte/s
    > ###################### #################### #### Ttl: 344.70 GByte
    >
    > With two copies I get up to 830-850MiB/s:
    >
    > Device eth0 [10.0.1.3] (1/1):
    > ================================================================================
    >
    > Incoming:
    > ############################################ ####
    > ############################################ ####
    > ############################################ ####
    > ############################################ ####
    > ############################################ ####
    > ############################################ #### Curr: 846.61 MByte/s
    > ############################################ #### Avg: 683.14 MByte/s
    > ############################################ #### Min: 0.00 MByte/s
    > ############################################ #### Max: 860.17 MByte/s
    > ############################################ #### Ttl: 305.71 GByte
    >
    > Using a 4MiB r/w size with NFS improves performance to sustain > 750MiB/s
    > a little better I think:
    > 10.0.1.4:/r1 /nfs/box2/r1 nfs
    > tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=4194304,wsize=4194304 0

    What about using UDP ?

    > Anyhow, roughly 750-850MiB/s it would be nice to get 1Gbyte/sec but I guess
    > the kerrnel (or my HW, CPU not fast enough) is not there yet.
    >
    > Also found a good doc from RedHat:
    > http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf
    >
    >
    > Justin.
    >

    HTH, Z.


    \
     
     \ /
      Last update: 2011-02-06 17:59    [W:0.028 / U:30.528 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site