lkml.org 
[lkml]   [1999]   [Mar]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectTCP window updates [Was: PROBLEM: Sending mail-attachment]


    Some people already reported problems with tcp stalls (Netscape,
    XEmacs/Gnus etc.). I did some debugging and hopefully found out,
    when and why this happens.
    Here again is a typical trace:

    > 13:53:20.339402 lh.3333 > lh.1284: P 25161:26641(1480) ack 1 win 31072 (DF)
    > 13:53:20.339402 lh.3333 > lh.1284: P 25161:26641(1480) ack 1 win 31072 (DF)
    > 13:53:20.339450 lh.1284 > lh.3333: . ack 26641 win 4432 (DF)
    > 13:53:20.339450 lh.1284 > lh.3333: . ack 26641 win 4432 (DF)
    > 13:53:20.339864 lh.3333 > lh.1284: P 26641:28121(1480) ack 1 win 31072 (DF)
    > 13:53:20.339864 lh.3333 > lh.1284: P 26641:28121(1480) ack 1 win 31072 (DF)
    > 13:53:20.339903 lh.1284 > lh.3333: . ack 28121 win 2952 (DF)
    > 13:53:20.339903 lh.1284 > lh.3333: . ack 28121 win 2952 (DF)
    > 13:53:20.340299 lh.3333 > lh.1284: P 28121:29601(1480) ack 1 win 31072 (DF)
    > 13:53:20.340299 lh.3333 > lh.1284: P 28121:29601(1480) ack 1 win 31072 (DF)
    > 13:53:20.340337 lh.1284 > lh.3333: . ack 29601 win 1472 (DF)
    > 13:53:20.340337 lh.1284 > lh.3333: . ack 29601 win 1472 (DF)
    *** stall ***

    As Stanislav Meduna already postet, one part of the problem is in
    tcp.c cleanup_rbuf():

    if((copied >= rcv_window_now) &&
    ((rcv_window_now + tp->mss_cache) <= tp->window_clamp))
    tcp_read_wakeup(sk);

    The first check should prevent unnecessary ACKs. Unfortunatly
    it also prevents advertising an increased window sometimes.
    When the reader always reads small chunks of data, say 15 bytes,
    the check will always fail since "copied" is always 15 while
    rcv_window_now mostly is larger.

    But why does the sender not simply send 1472 bytes after receiving
    13:53:20.340337 lh.1284 > lh.3333: . ack 29601 win 1472 (DF) ?
    The reader would then advertise win 0 and everything would be fine
    (the sender would use TIME_PROBE0 to frequently probe the win size
    of the reader).
    The reason is that the SendQ is empty. This is the timing issue here
    which made the problem appear like a race.

    Since SendQ is empty now, the kernel will wake up the sender proc.
    The sender now continues calling write(). This invokes tcp_v4_sendmsg()
    in tcp_ipv4.c. Now there are 2 possibilities that data actually gets
    transmitted.

    1. In tcp_v4_sendmsg() -> tcp_do_sendmsg() ->
    tcp_send_skb((sk, skb, queue_it);

    For small write sizes ( < mss and < max_window/2 ) "queue_it"
    is always true. This prevents any transmitting and only queues
    the data.

    2. In tcp_v4_sendmsg() ->

    if(tp->send_head && tcp_snd_test(sk, tp->send_head))
    tcp_write_xmit(sk);

    The same. tcp_snd_test will always fail for small write() sizes.


    ** Summary **

    A tcp stall occures when:

    1. Reader sends an ACK advertising a win < mss / 2.
    2. Senders SendQ is empty in this moment.
    3. Senders subsequent write() calls use write sizes < mss.


    The attached program can demonstrate that. It creates a sender
    and a reader. The sender sends about 50kb to the reader while
    the reader sleeps().

    netstat says correctly:

    Proto Recv-Q Send-Q Local Address Foreign Address State
    tcp 0 0 localhost:3334 localhost:1033 ESTABLISHED
    tcp 57500 0 localhost:1033 localhost:3334 ESTABLISHED


    Now the reader wakes up and reads the data and the sender sleeps()
    10 seconds. Then the sender writes again 3871 bytes, which is the
    critical size on my machine (mtu 3924).

    Proto Recv-Q Send-Q Local Address Foreign Address State
    tcp 0 3872 localhost:3334 localhost:1033 FIN_WAIT1
    tcp 0 0 localhost:1033 localhost:3334 ESTABLISHED

    There is the stall. No further write() will cause any data transmission
    as long as the write size is <= 3871.


    I can not say whether sender or reader side is wrong. I would say both
    but I still have to read the RFCs :).


    Any comments?


    -- Matthias
    #include <stdio.h>
    #include <string.h>
    #include <stdlib.h>
    #include <sys/time.h>
    #include <sys/types.h>
    #include <unistd.h>
    #include <netinet/in.h>
    #include <sys/un.h>
    #include <errno.h>
    #include <fcntl.h>
    #include <sys/wait.h>
    #include <signal.h>
    #include <netdb.h>

    #define WSIZE1 57500
    #define WSIZE2 3871
    #define RSIZE 15
    #define PORT 3334

    char buffer[WSIZE1 + WSIZE2];

    void server(void)
    {
    struct sockaddr_in addr, accept_addr;
    int addrlen = 0, mainSock, sock;

    addr.sin_family = AF_INET;
    addr.sin_port = htons(PORT);
    addr.sin_addr.s_addr = INADDR_ANY;

    if((mainSock = socket(addr.sin_family, SOCK_STREAM, 0)) < 0)
    { perror("socket()"); exit(1); }

    if( bind(mainSock, &addr, sizeof(addr)) < 0)
    { perror("bind()"); exit(1); }

    if( listen(mainSock, 1) < 0)
    { perror("listen()"); exit(1); }

    sock = accept(mainSock, (struct sockaddr *)&accept_addr, &addrlen);
    if(sock < 0) { perror("accept()"); abort(); }

    write(sock, buffer, WSIZE1);

    sleep(10);

    write(sock, buffer, WSIZE2);

    sleep(3);
    close(sock);
    close(mainSock);
    abort();
    }

    static void client(void)
    {
    struct sockaddr_in addr;
    struct hostent *hostent;
    char buf[RSIZE];
    int len, sock;

    if(!(hostent = gethostbyname("localhost")))
    { printf("Could not resolve host!\n"); exit(1); }

    addr.sin_port = htons(PORT);
    addr.sin_family = hostent->h_addrtype;
    memcpy(&addr.sin_addr, hostent->h_addr, hostent->h_length);

    if((sock = socket(addr.sin_family, SOCK_STREAM, 0)) < 0)
    { perror("socket()"); exit(1); }

    while(connect(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0)
    { perror("connect()"); exit(1); }

    sleep(5);

    len = 0;
    while(len < WSIZE1 + WSIZE2)
    len += read(sock, buf, RSIZE);

    close(sock);
    }

    int main(int argc, char **argv)
    {
    int i;

    i = fork();
    if(!i)
    server();

    sleep(2);
    client();
    wait(&i);

    return 0;
    }
    \
     
     \ /
      Last update: 2005-03-22 13:50    [W:0.032 / U:0.548 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site