lkml.org 
[lkml]   [2006]   [Nov]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectZONE_NORMAL memory exhausted by 4000 TCP sockets
    Dears,
    I'm running a linux box with kernel version 2.6.16. The hardware
    has 2 Woodcrest Xeon CPUs (2 cores each) and 4G RAM. The NIC cards is
    Intel 82571 on PCI-e bus.
    The box is acting as ethernet bridge between 2 Gigabit Ethernets.
    By configuring ebtables and iptables, an application is running as TCP
    proxy which will intercept all TCP connections requests from the
    network and setup another TCP connection to the acture server. The
    TCP proxy then relays all traffics in both directions.
    The problem is the memory. Since the box must support thousands of
    concurrent connections, I know the memory size of ZONE_NORMAL would be
    a bottleneck as TCP packets would need many buffers. After setting
    upper limit of net.ipv4.tcp_rmem and net.ipv4.tcp_wmem to 32K bytes,
    our test began.
    My test scenario employs 2000 concurrent downloading connections
    to a IIS server's port 80. The throughput is about 500~600 Mbps which
    is limited by the capability of the client application. Because all
    traffics are from server to client and the capability of client
    machine is bottleneck, I believe the receiver side of the sockets
    connected with server and the sender side of the sockets connected
    with client should be filled with packets in correspondent windows.
    Thus, roughly there should be about 32K * 2000+ 32K*2000 = 128M bytes
    memory occupied by TCP/IP stack for packet buffering. Data from
    slabtop confermed it. it's about 140M bytes memory cost after I start
    the traffic. That reasonablly matched with my estimation. However,
    /proc/meminfo had a different story. The 'LowFree' dropped from about
    710M to 80M. In other words, there's addtional 500M memory in
    ZONE_NORMAL allocated by someone other than the slab. Why?
    I also made another test that the upper limit of tcp_rmem and
    tcp_wmem being set to 64K. After 2000 connections transfering a lot of
    data for several seconds, the linux box showed some error messages
    such as error allocating memory pages, etc. and became unstable.
    My questions are:

    1. To calculate memory request of TCP sockets, is there any other
    large amount of memory requested besides send and receive buffer?
    2. Is there any logics that emploied by TCP/IP stack that will
    dynamically allocating memory pages directly instead of from slab?

    Thanks!

    Xiaoming.
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2006-11-06 07:09    [W:3.189 / U:0.412 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site