Messages in this thread | | | Date | Sat, 15 Nov 2008 02:29:27 -0500 | From | "Karl Pickett" <> | Subject | Re: tcp_tw_recycle broken? |
| |
On Sat, Nov 15, 2008 at 12:57 AM, Willy Tarreau <w@1wt.eu> wrote: > On Fri, Nov 14, 2008 at 11:37:06PM -0500, Karl Pickett wrote: >> Hey. Developing a http proxy on fedora 9 (2.6.25) and running into a >> strange issue. >> >> Having the proxy set up and tear down 6000 tcp connections a second to >> the same test server ip and port, >> it quickly blows up (5 seconds) due to all 30000 ephemeral ports going >> to TIME_WAIT. >> setting tw_recycle=1 fixed the problem, and there are never more than >> a couple hundred ports in TIME_WAIT. >> >> BUT... >> >> Changing the load test to alternate between two test server ips, it >> blows up. Connect: can't assign requested address. (note I am not >> binding before hand, I tried >> and binding first to port 0 made no difference - it just blows up then >> during the bind). >> >> And there are ~28K ports in TIME_WAIT. For example: >> >> proxy_ip:30000 load_test_1:8080 TIME_WAIT >> proxy_ip:30000 load_test_2:8080 TIME_WAIT >> ... >> but most are not duplicates of the same local port. >> >> >> What. The. Heck. >> >> So short of rebuilding the kernel with time_wait as 1 second, is there >> any other way not to brick my proxy? > > two things : > - set tcp_tw_reuse to 1 too. > - do a setsockopt(SO_REUSEADDR) before connect() > > Using this, my proxy has no problem at 35K sess/s on 2.6.25. I'm not sure > if disabling either option above still works. > > Hoping this helps, > Willy > >
Thanks for the help. Well, it looks like tw_reuse is what I wanted... not tw_recycle. Based on a python test program over loopback, tw_reuse alone solves the problem... so_reuseaddr doesn't do anything. And apparently the tcp code is too much for me...looking at the source I thought tw_reuse only can happen when timestamps are enabled. But even after disabling timestamps tw_reuse still works over loopback.
I'll have to wait until Monday to try it again in the lab. I was trying combinations of tw_reuse and recycle, too many to remember apparently.
May I just confirm.. is tcp_tw_reuse NOT dependent on receiving timestamps?
-- Karl Pickett
| |