lkml.org 
[lkml]   [2008]   [Nov]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: tcp_tw_recycle broken?
On Sat, Nov 15, 2008 at 12:57 AM, Willy Tarreau <w@1wt.eu> wrote:
> On Fri, Nov 14, 2008 at 11:37:06PM -0500, Karl Pickett wrote:
>> Hey. Developing a http proxy on fedora 9 (2.6.25) and running into a
>> strange issue.
>>
>> Having the proxy set up and tear down 6000 tcp connections a second to
>> the same test server ip and port,
>> it quickly blows up (5 seconds) due to all 30000 ephemeral ports going
>> to TIME_WAIT.
>> setting tw_recycle=1 fixed the problem, and there are never more than
>> a couple hundred ports in TIME_WAIT.
>>
>> BUT...
>>
>> Changing the load test to alternate between two test server ips, it
>> blows up. Connect: can't assign requested address. (note I am not
>> binding before hand, I tried
>> and binding first to port 0 made no difference - it just blows up then
>> during the bind).
>>
>> And there are ~28K ports in TIME_WAIT. For example:
>>
>> proxy_ip:30000 load_test_1:8080 TIME_WAIT
>> proxy_ip:30000 load_test_2:8080 TIME_WAIT
>> ...
>> but most are not duplicates of the same local port.
>>
>>
>> What. The. Heck.
>>
>> So short of rebuilding the kernel with time_wait as 1 second, is there
>> any other way not to brick my proxy?
>
> two things :
> - set tcp_tw_reuse to 1 too.
> - do a setsockopt(SO_REUSEADDR) before connect()
>
> Using this, my proxy has no problem at 35K sess/s on 2.6.25. I'm not sure
> if disabling either option above still works.
>
> Hoping this helps,
> Willy
>
>

Thanks for the help. Well, it looks like tw_reuse is what I wanted...
not tw_recycle. Based on a python test program over loopback,
tw_reuse alone solves the problem... so_reuseaddr doesn't do anything.
And apparently the tcp code is too much for me...looking at the
source I thought tw_reuse only can happen when timestamps are enabled.
But even after disabling timestamps tw_reuse still works over
loopback.

I'll have to wait until Monday to try it again in the lab. I was
trying combinations of tw_reuse and recycle, too many to remember
apparently.

May I just confirm.. is tcp_tw_reuse NOT dependent on receiving timestamps?


--
Karl Pickett


\
 
 \ /
  Last update: 2008-11-15 08:31    [W:0.050 / U:0.240 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site