Messages in this thread | | | Date | Mon, 2 Jun 2008 11:23:53 +0200 | From | Ingo Molnar <> | Subject | Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+ |
| |
* Eric Dumazet <dada1@cosmosbay.com> wrote:
>>> 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62 >>> 22003 listen(4, 10) = 0 >>> 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0 >>> >>> i'll queue up your reverts for testing in -tip.
I turned off localhost distcc two days ago and there has not been a single hung socket since then, so we now know it for sure that without localhost distcc connections, -tip's QA will not produce any hung sockets in about 1000 random-kernel-build+boot iterations.
i've added those reverts this morning and added back the localhost distcc rules - we'll see whether the hung sockets are back.
> I believe Ingo problems come on long lived sockets (were many bytes > were exchanged between the peers), so I dont think DEFER_ACCEPT is the > cullprit. > > I suggest to enable CONFIG_TIMER_STATS and to check timers, because > /proc/net/tcp can display apparently large timer values when the timer > is elapsed (jiffies > icsk->icsk_timeout) and > jiffies_to_clock_t(timer_expires - jiffies) is then overflowing doing > a multiply and a divide.
i'm wondering whether your suspicion on broken TCP timers is consistent with the symptoms i've seen: the hung sockets clearly produced periodic packet activity every 180 seconds, up to 8 hours, without ever changing their receive of send queue. So at least a part of the TCP timer mechanism for that specific stuck socket was working fine.
is there no sysctl or other debug mechanism to somehow get its full TCP state and the reasons for why it is stuck? I'm wondering how you debug broken TCP state machines without enabling testers to be able to dump all state and passing it to developers.
I have a clearly reproducable testcase and i'd like to help out, but the whole effort is stalled on 'not enough information' it appears. Doing random reverts might help in truly helpless situations where a bug has no debuggable state - but this situation seems really routine to me: it's very difficult to trigger the bug but once it triggers the bug scenario is stable and analyzable. I'd be glad to test any instrumentation patch that makes similar scenarios more analyzable.
Ingo
| |