Messages in this thread |  | | Subject | Re: Resource starvation on a 2.2.19 web server | Date | Tue, 11 Sep 2001 15:48:03 +0100 (BST) | From | Alan Cox <> |
| |
> > > I'm not on the linux-kernel list; please specifically include me in > any reply. Thanks... > > I'm seeing a problem with what appears to be running out of file (or > inode) handles on an apache web server. I've done everything I have > been able to find to fix this but have run out of things to try. > The summary below is as concise and complete as I can make it; if > additional information is needed I'll happily provide it. I've run > linux since 1.57, but I've never had to dig into tuning at this > level until now. > > This is a 1GB dual PIII-650 box. I'm using the Mandrake 2.2.19-4.3 > source tree, their latest, as a base for the kernel. The server is > running a couple of software raid partitions and a 26 GB EATA SCSI > partition. Most of the file systems are ext2, with a Reiser fs on > the SCSI partition. > > Based on logged errors, some time ago I started increasing > /proc/sys/fs/{file,inode}-max to 98304 and 524300 respectively. > The large counts are pushed by rsync in syncing the large web tree > onto another box. > > >From what I have found on the 'net on kernel tuning, I have made a > custom 2.2.19 kernel with the following changes in the include > files. I am not seeing logged errors on any sort of resource > starvation on this kernel; before upping the limits I was seeing > both system and apache errors about no sockets and file handles. > > limits.h #define NR_OPEN 2048 (overridded by file-nr, I think) > limits.h #define OPEN_MAX 2048 (max open files per process) > posix_types.h: #define __FD_SETSIZE 2048 > socket.h: #define SOMAXCONN 16384 > > The symptom is that after the box runs for a while, nslookup cannot > resolve a name; it complains that it cannot find the configured DNS > servers. If I USR1 warm start apache, nslookup immediately works > again. Some system resource is being freed up when apache is > restarted, but I've been unable to isolate what it is or fix it. > > The two snapshots below are of /proc/sys/fs/* before and after an > apache warm start. There is some strange info in file-nr and > inode-nr. Per linux/Documentation/proc.txt, the middle number of > file-nr is "the number of used file handles", and the middle number > of inode-nr is "the number of free inodes". > > After reset, the number of free inodes goes up. What seems very > strange is that the number of free files goes from 4 to 57034, > almost as if beyond a certain limit the file-nr counter starts > acting as an underflow counter (just my own observation). > Particularly inconsistent is that the number of allocated file > handles before warm start, 58783, is well below the permitted > maximum of 98304. > > Has anyone seen anything like this to help me understand where to go > next? Thanks much in advance, > > -- Michael > > > # before apache warm start > 0 421083 45 0 0 0 > 1024 > 0 0 > 98304 > 58783 4 98304 <=== file-nr: note the middle number > 524288 > 524300 43622 <=== inode-nr: note the middle number > 524300 43622 0 0 0 0 0 > 256 > 12 > > # after apache warm start > 0 421625 45 0 0 0 > 1024 > 0 0 > 98304 > 58797 57034 98304 <=== was 4 before warm start > 524288 > 524300 99808 <=== was 43622 before warm start > 524300 99808 0 0 0 0 0 > 256 > 12 > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ >
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
|  |