lkml.org 
[lkml]   [2006]   [Oct]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Dropping NETIF_F_SG since no checksum feature.
    Quoting r. David Miller <davem@davemloft.net>:
    > Subject: Re: Dropping NETIF_F_SG since no checksum feature.
    >
    > From: "Michael S. Tsirkin" <mst@mellanox.co.il>
    > Date: Wed, 11 Oct 2006 23:23:39 +0200
    >
    > > With my patch, there is a huge performance gain by increasing MTU to 64K.
    > > And it seems the only way to do this is by S/G.
    >
    > Numbers?
    >

    I created two subnets on top of the same pair infiniband HCAs:

    root@sw069 ~]# ifconfig ib0
    ib0 Link encap:UNSPEC HWaddr
    00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
    inet addr:12.4.3.69 Bcast:12.255.255.255 Mask:255.0.0.0
    inet6 addr: fe80::202:c902:20:ee45/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
    RX packets:1382531 errors:0 dropped:0 overruns:0 frame:0
    TX packets:2725206 errors:0 dropped:5 overruns:0 carrier:0
    collisions:0 txqueuelen:128
    RX bytes:71892772 (68.5 MiB) TX bytes:5290011992 (4.9 GiB)

    [root@sw069 ~]# ifconfig ibc0
    ibc0 Link encap:UNSPEC HWaddr
    00-03-04-06-FE-80-00-00-00-00-00-00-00-00-00-00
    inet addr:11.4.3.69 Bcast:11.255.255.255 Mask:255.0.0.0
    inet6 addr: fe80::202:c902:20:ee45/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:65484 Metric:1
    RX packets:115647 errors:0 dropped:0 overruns:0 frame:0
    TX packets:253403 errors:0 dropped:4 overruns:0 carrier:0
    collisions:0 txqueuelen:128
    RX bytes:6014720 (5.7 MiB) TX bytes:16589589008 (15.4 GiB)

    The other side was configured with 12.4.3.68 for MTU 65484
    and 11.4.3.68 for MTU 2044.

    And then I just run netperf:
    [root@sw069 ~]#
    [root@sw069 ~]# /mswg/work/mst/netperf-2.4.2/src/netperf -f M -H 12.4.3.68 -c -C
    TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 12.4.3.68 (12.4.3.68)
    port 0 AF_INET
    Recv Send Send Utilization Service Demand
    Socket Socket Message Elapsed Send Recv Send Recv
    Size Size Size Time Throughput local remote local remote
    bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB

    87380 16384 16384 10.00 286.45 40.20 25.28 5.482 3.448

    [root@sw069 ~]# /mswg/work/mst/netperf-2.4.2/src/netperf -f M -H 11.4.3.68
    TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.68 (11.4.3.68)
    port 0 AF_INET
    Recv Send Send
    Socket Socket Message Elapsed
    Size Size Size Time Throughput
    bytes bytes bytes secs. MBytes/sec

    87380 16384 16384 10.01 782.55

    This is all very preliminary - but I hope you get the idea -
    increasing MTU is very helpful for infiniband, and infiniband adapters
    handle large S/G lists without problems, but the verbs
    do not include support for IP checksums, so these must be done in software.

    So what we would like, is for the infiniband network device to say
    "I don't support checksums, I only support S/G" and then for
    network layer to do the checksumming for us piggybacking on data copy
    at least for cases where it does perform the copy.

    Does this makes sense now?

    --
    MST
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2006-10-12 21:15    [W:0.024 / U:67.752 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site