lkml.org 
[lkml]   [1999]   [Mar]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectTCP window updates [Was: PROBLEM: Sending mail-attachment]


Some people already reported problems with tcp stalls (Netscape,
XEmacs/Gnus etc.). I did some debugging and hopefully found out,
when and why this happens.
Here again is a typical trace:

> 13:53:20.339402 lh.3333 > lh.1284: P 25161:26641(1480) ack 1 win 31072 (DF)
> 13:53:20.339402 lh.3333 > lh.1284: P 25161:26641(1480) ack 1 win 31072 (DF)
> 13:53:20.339450 lh.1284 > lh.3333: . ack 26641 win 4432 (DF)
> 13:53:20.339450 lh.1284 > lh.3333: . ack 26641 win 4432 (DF)
> 13:53:20.339864 lh.3333 > lh.1284: P 26641:28121(1480) ack 1 win 31072 (DF)
> 13:53:20.339864 lh.3333 > lh.1284: P 26641:28121(1480) ack 1 win 31072 (DF)
> 13:53:20.339903 lh.1284 > lh.3333: . ack 28121 win 2952 (DF)
> 13:53:20.339903 lh.1284 > lh.3333: . ack 28121 win 2952 (DF)
> 13:53:20.340299 lh.3333 > lh.1284: P 28121:29601(1480) ack 1 win 31072 (DF)
> 13:53:20.340299 lh.3333 > lh.1284: P 28121:29601(1480) ack 1 win 31072 (DF)
> 13:53:20.340337 lh.1284 > lh.3333: . ack 29601 win 1472 (DF)
> 13:53:20.340337 lh.1284 > lh.3333: . ack 29601 win 1472 (DF)
*** stall ***

As Stanislav Meduna already postet, one part of the problem is in
tcp.c cleanup_rbuf():

if((copied >= rcv_window_now) &&
((rcv_window_now + tp->mss_cache) <= tp->window_clamp))
tcp_read_wakeup(sk);
The first check should prevent unnecessary ACKs. Unfortunatly
it also prevents advertising an increased window sometimes.
When the reader always reads small chunks of data, say 15 bytes,
the check will always fail since "copied" is always 15 while
rcv_window_now mostly is larger.

But why does the sender not simply send 1472 bytes after receiving
13:53:20.340337 lh.1284 > lh.3333: . ack 29601 win 1472 (DF) ?
The reader would then advertise win 0 and everything would be fine
(the sender would use TIME_PROBE0 to frequently probe the win size
of the reader).
The reason is that the SendQ is empty. This is the timing issue here
which made the problem appear like a race.
Since SendQ is empty now, the kernel will wake up the sender proc.
The sender now continues calling write(). This invokes tcp_v4_sendmsg()
in tcp_ipv4.c. Now there are 2 possibilities that data actually gets
transmitted.

1. In tcp_v4_sendmsg() -> tcp_do_sendmsg() ->
tcp_send_skb((sk, skb, queue_it);
For small write sizes ( < mss and < max_window/2 ) "queue_it"
is always true. This prevents any transmitting and only queues
the data.

2. In tcp_v4_sendmsg() ->
if(tp->send_head && tcp_snd_test(sk, tp->send_head))
tcp_write_xmit(sk);
The same. tcp_snd_test will always fail for small write() sizes.


** Summary **

A tcp stall occures when:

1. Reader sends an ACK advertising a win < mss / 2.
2. Senders SendQ is empty in this moment.
3. Senders subsequent write() calls use write sizes < mss.


The attached program can demonstrate that. It creates a sender
and a reader. The sender sends about 50kb to the reader while
the reader sleeps().

netstat says correctly:

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:3334 localhost:1033 ESTABLISHED
tcp 57500 0 localhost:1033 localhost:3334 ESTABLISHED


Now the reader wakes up and reads the data and the sender sleeps()
10 seconds. Then the sender writes again 3871 bytes, which is the
critical size on my machine (mtu 3924).

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 3872 localhost:3334 localhost:1033 FIN_WAIT1
tcp 0 0 localhost:1033 localhost:3334 ESTABLISHED

There is the stall. No further write() will cause any data transmission
as long as the write size is <= 3871.


I can not say whether sender or reader side is wrong. I would say both
but I still have to read the RFCs :).


Any comments?


-- Matthias
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#include <netinet/in.h>
#include <sys/un.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/wait.h>
#include <signal.h>
#include <netdb.h>
#define WSIZE1 57500
#define WSIZE2 3871
#define RSIZE 15
#define PORT 3334

char buffer[WSIZE1 + WSIZE2];
void server(void)
{
struct sockaddr_in addr, accept_addr;
int addrlen = 0, mainSock, sock;
addr.sin_family = AF_INET;
addr.sin_port = htons(PORT);
addr.sin_addr.s_addr = INADDR_ANY;
if((mainSock = socket(addr.sin_family, SOCK_STREAM, 0)) < 0)
{ perror("socket()"); exit(1); }
if( bind(mainSock, &addr, sizeof(addr)) < 0)
{ perror("bind()"); exit(1); }
if( listen(mainSock, 1) < 0)
{ perror("listen()"); exit(1); }
sock = accept(mainSock, (struct sockaddr *)&accept_addr, &addrlen);
if(sock < 0) { perror("accept()"); abort(); }
write(sock, buffer, WSIZE1);
sleep(10);
write(sock, buffer, WSIZE2);
sleep(3);
close(sock);
close(mainSock);
abort();
}
static void client(void)
{
struct sockaddr_in addr;
struct hostent *hostent;
char buf[RSIZE];
int len, sock;
if(!(hostent = gethostbyname("localhost")))
{ printf("Could not resolve host!\n"); exit(1); }
addr.sin_port = htons(PORT);
addr.sin_family = hostent->h_addrtype;
memcpy(&addr.sin_addr, hostent->h_addr, hostent->h_length);
if((sock = socket(addr.sin_family, SOCK_STREAM, 0)) < 0)
{ perror("socket()"); exit(1); }
while(connect(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0)
{ perror("connect()"); exit(1); }
sleep(5);
len = 0;
while(len < WSIZE1 + WSIZE2)
len += read(sock, buf, RSIZE);
close(sock);
}
int main(int argc, char **argv)
{
int i;
i = fork();
if(!i)
server();
sleep(2);
client();
wait(&i);
return 0;
}
\
 
 \ /
  Last update: 2005-03-22 13:50    [from the cache]
©2003-2011 Jasper Spaans