[lkml]   [2008]   [Jan]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: e1000 full-duplex TCP performance well below wire speed
Hi Sangtae,

Thanks for joining this discussion -- it's good to a CUBIC author and
expert here!

>> In our application (cluster computing) we use a very tightly coupled
>> high-speed low-latency network. There is no 'wide area traffic'. So
>> it's hard for me to understand why any networking components or
>> software layers should take more than milliseconds to ramp up or back
>> off in speed. Perhaps we should be asking for a TCP congestion
>> avoidance algorithm which is designed for a data center environment
>> where there are very few hops and typical packet delivery times are
>> tens or hundreds of microseconds. It's very different than delivering
>> data thousands of km across a WAN.

> If your network latency is low, regardless of type of protocols should
> give you more than 900Mbps.

Yes, this is also what I had thought.

In the graph that we posted, the two machines are connected by an ethernet
crossover cable. The total RTT of the two machines is probably AT MOST a
couple of hundred microseconds. Typically it takes 20 or 30 microseconds
to get the first packet out the NIC. Travel across the wire is a few
nanoseconds. Then getting the packet into the receiving NIC might be
another 20 or 30 microseconds. The ACK should fly back in about the same

> I can guess the RTT of two machines is less than 4ms in your case and I
> remember the throughputs of all high-speed protocols (including
> tcp-reno) were more than 900Mbps with 4ms RTT. So, my question which
> kernel version did you use with your broadcomm NIC and got more than
> 900Mbps?

We are going to double-check this (we did the broadcom testing about two
months ago). Carsten is going to re-run the broadcomm experiments later
today and will then post the results.

You can see results from some testing on crossover-cable wired systems
with broadcomm NICs, that I did about 2 years ago, here:
You'll notice that total TCP throughput on the crossover cable was about
220 MB/sec. With TCP overhead this is very close to 2Gb/s.

> I have two machines connected by a gig switch and I can see what happens
> in my environment. Could you post what parameters did you use for
> netperf testing?

Carsten will post these in the next few hours. If you want to simplify
further, you can even take away the gig switch and just use a crossover

> and also if you set any parameters for your testing, please post them
> here so that I can see that happens to me as well.

Carsten will post all the sysctl and ethtool parameters shortly.

Thanks again for chiming in. I am sure that with help from you, Jesse, and
Rick, we can figure out what is going on here, and get it fixed.


 \ /
  Last update: 2008-01-31 09:55    [W:0.193 / U:0.396 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site