Messages in this thread | | | Date | Mon, 6 Nov 2006 16:10:03 +0800 | From | "Zhao Xiaoming" <> | Subject | Re: ZONE_NORMAL memory exhausted by 4000 TCP sockets |
| |
2006/11/6, Eric Dumazet <dada1@cosmosbay.com>: > Zhao Xiaoming a écrit : > > Dears, > > I'm running a linux box with kernel version 2.6.16. The hardware > > has 2 Woodcrest Xeon CPUs (2 cores each) and 4G RAM. The NIC cards is > > Intel 82571 on PCI-e bus. > > The box is acting as ethernet bridge between 2 Gigabit Ethernets. > > By configuring ebtables and iptables, an application is running as TCP > > proxy which will intercept all TCP connections requests from the > > network and setup another TCP connection to the acture server. The > > TCP proxy then relays all traffics in both directions. > > The problem is the memory. Since the box must support thousands of > > concurrent connections, I know the memory size of ZONE_NORMAL would be > > a bottleneck as TCP packets would need many buffers. After setting > > upper limit of net.ipv4.tcp_rmem and net.ipv4.tcp_wmem to 32K bytes, > > our test began. > > My test scenario employs 2000 concurrent downloading connections > > to a IIS server's port 80. The throughput is about 500~600 Mbps which > > is limited by the capability of the client application. Because all > > traffics are from server to client and the capability of client > > machine is bottleneck, I believe the receiver side of the sockets > > connected with server and the sender side of the sockets connected > > with client should be filled with packets in correspondent windows. > > Thus, roughly there should be about 32K * 2000+ 32K*2000 = 128M bytes > > memory occupied by TCP/IP stack for packet buffering. Data from > > slabtop confermed it. it's about 140M bytes memory cost after I start > > the traffic. That reasonablly matched with my estimation. However, > > /proc/meminfo had a different story. The 'LowFree' dropped from about > > 710M to 80M. In other words, there's addtional 500M memory in > > ZONE_NORMAL allocated by someone other than the slab. Why? > > We dont know. You might post some data so that we can have some ideas. > > Also, these kind of question is better handled by linux netdev mailing list, > so I added a CC to this list. > > cat /proc/slabinfo > cat /proc/meminfo > cat /proc/net/sockstat > cat /proc/buddyinfo > > > I also made another test that the upper limit of tcp_rmem and > > tcp_wmem being set to 64K. After 2000 connections transfering a lot of > > data for several seconds, the linux box showed some error messages > > such as error allocating memory pages, etc. and became unstable. > > My questions are: > > > > 1. To calculate memory request of TCP sockets, is there any other > > large amount of memory requested besides send and receive buffer? > > 2. Is there any logics that emploied by TCP/IP stack that will > > dynamically allocating memory pages directly instead of from slab? > > TCP stack is one thing, but other things may consume ram on your kernel. > > Also, kernel memory allocation might use twice the ram you intend to use > because of power of two alignments. > > Are you using iptables connection tracking ? > > If you plan to use a lot of RAM in kernel, why dont you use a 64 bits kernel, > so that all ram is available for kernel, not only 900 MB ? > > Eric > > Thanks for the answer. I know it's more likely relats to netdev. However, it's always a strange thing to have 400~500M bytes LOWMEM 'gone' while it's not reported to be occupied by slab. Both meminfo and buddyinfo tell the same.
with traffics of 2000 concurrent sessions:
cat /proc/meminfo MemTotal: 4136580 kB MemFree: 3298460 kB Buffers: 4096 kB Cached: 21124 kB SwapCached: 0 kB Active: 47416 kB Inactive: 12532 kB HighTotal: 3276160 kB HighFree: 3214592 kB LowTotal: 860420 kB LowFree: 83868 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 12 kB Writeback: 0 kB Mapped: 42104 kB Slab: 293952 kB CommitLimit: 2068288 kB Committed_AS: 58892 kB PageTables: 1112 kB VmallocTotal: 116728 kB VmallocUsed: 2940 kB VmallocChunk: 110548 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB
without traffics:
cat /proc/meminfo MemTotal: 4136580 kB MemFree: 3924276 kB Buffers: 4460 kB Cached: 21020 kB SwapCached: 0 kB Active: 47592 kB Inactive: 12848 kB HighTotal: 3276160 kB HighFree: 3214716 kB LowTotal: 860420 kB LowFree: 709560 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 40 kB Writeback: 0 kB Mapped: 42368 kB Slab: 132172 kB CommitLimit: 2068288 kB Committed_AS: 59220 kB PageTables: 1140 kB VmallocTotal: 116728 kB VmallocUsed: 2940 kB VmallocChunk: 110548 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB
Xiaoming. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |