Messages in this thread | | | Date | Thu, 31 Jan 2008 02:52:13 -0600 (CST) | From | Bruce Allen <> | Subject | Re: e1000 full-duplex TCP performance well below wire speed |
| |
Hi Sangtae,
Thanks for joining this discussion -- it's good to a CUBIC author and expert here!
>> In our application (cluster computing) we use a very tightly coupled >> high-speed low-latency network. There is no 'wide area traffic'. So >> it's hard for me to understand why any networking components or >> software layers should take more than milliseconds to ramp up or back >> off in speed. Perhaps we should be asking for a TCP congestion >> avoidance algorithm which is designed for a data center environment >> where there are very few hops and typical packet delivery times are >> tens or hundreds of microseconds. It's very different than delivering >> data thousands of km across a WAN.
> If your network latency is low, regardless of type of protocols should > give you more than 900Mbps.
Yes, this is also what I had thought.
In the graph that we posted, the two machines are connected by an ethernet crossover cable. The total RTT of the two machines is probably AT MOST a couple of hundred microseconds. Typically it takes 20 or 30 microseconds to get the first packet out the NIC. Travel across the wire is a few nanoseconds. Then getting the packet into the receiving NIC might be another 20 or 30 microseconds. The ACK should fly back in about the same time.
> I can guess the RTT of two machines is less than 4ms in your case and I > remember the throughputs of all high-speed protocols (including > tcp-reno) were more than 900Mbps with 4ms RTT. So, my question which > kernel version did you use with your broadcomm NIC and got more than > 900Mbps?
We are going to double-check this (we did the broadcom testing about two months ago). Carsten is going to re-run the broadcomm experiments later today and will then post the results.
You can see results from some testing on crossover-cable wired systems with broadcomm NICs, that I did about 2 years ago, here: http://www.lsc-group.phys.uwm.edu/beowulf/nemo/design/SMC_8508T_Performance.html You'll notice that total TCP throughput on the crossover cable was about 220 MB/sec. With TCP overhead this is very close to 2Gb/s.
> I have two machines connected by a gig switch and I can see what happens > in my environment. Could you post what parameters did you use for > netperf testing?
Carsten will post these in the next few hours. If you want to simplify further, you can even take away the gig switch and just use a crossover cable.
> and also if you set any parameters for your testing, please post them > here so that I can see that happens to me as well.
Carsten will post all the sysctl and ethtool parameters shortly.
Thanks again for chiming in. I am sure that with help from you, Jesse, and Rick, we can figure out what is going on here, and get it fixed.
Cheers, Bruce
| |