lkml.org 
[lkml]   [2001]   [Sep]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectResource starvation on a 2.2.19 web server

    I'm not on the linux-kernel list; please specifically include me in
    any reply. Thanks...

    I'm seeing a problem with what appears to be running out of file (or
    inode) handles on an apache web server. I've done everything I have
    been able to find to fix this but have run out of things to try.
    The summary below is as concise and complete as I can make it; if
    additional information is needed I'll happily provide it. I've run
    linux since 1.57, but I've never had to dig into tuning at this
    level until now.

    This is a 1GB dual PIII-650 box. I'm using the Mandrake 2.2.19-4.3
    source tree, their latest, as a base for the kernel. The server is
    running a couple of software raid partitions and a 26 GB EATA SCSI
    partition. Most of the file systems are ext2, with a Reiser fs on
    the SCSI partition.

    Based on logged errors, some time ago I started increasing
    /proc/sys/fs/{file,inode}-max to 98304 and 524300 respectively.
    The large counts are pushed by rsync in syncing the large web tree
    onto another box.

    >From what I have found on the 'net on kernel tuning, I have made a
    custom 2.2.19 kernel with the following changes in the include
    files. I am not seeing logged errors on any sort of resource
    starvation on this kernel; before upping the limits I was seeing
    both system and apache errors about no sockets and file handles.

    limits.h #define NR_OPEN 2048 (overridded by file-nr, I think)
    limits.h #define OPEN_MAX 2048 (max open files per process)
    posix_types.h: #define __FD_SETSIZE 2048
    socket.h: #define SOMAXCONN 16384

    The symptom is that after the box runs for a while, nslookup cannot
    resolve a name; it complains that it cannot find the configured DNS
    servers. If I USR1 warm start apache, nslookup immediately works
    again. Some system resource is being freed up when apache is
    restarted, but I've been unable to isolate what it is or fix it.

    The two snapshots below are of /proc/sys/fs/* before and after an
    apache warm start. There is some strange info in file-nr and
    inode-nr. Per linux/Documentation/proc.txt, the middle number of
    file-nr is "the number of used file handles", and the middle number
    of inode-nr is "the number of free inodes".

    After reset, the number of free inodes goes up. What seems very
    strange is that the number of free files goes from 4 to 57034,
    almost as if beyond a certain limit the file-nr counter starts
    acting as an underflow counter (just my own observation).
    Particularly inconsistent is that the number of allocated file
    handles before warm start, 58783, is well below the permitted
    maximum of 98304.

    Has anyone seen anything like this to help me understand where to go
    next? Thanks much in advance,

    -- Michael


    # before apache warm start
    0 421083 45 0 0 0
    1024
    0 0
    98304
    58783 4 98304 <=== file-nr: note the middle number
    524288
    524300 43622 <=== inode-nr: note the middle number
    524300 43622 0 0 0 0 0
    256
    12

    # after apache warm start
    0 421625 45 0 0 0
    1024
    0 0
    98304
    58797 57034 98304 <=== was 4 before warm start
    524288
    524300 99808 <=== was 43622 before warm start
    524300 99808 0 0 0 0 0
    256
    12


    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2005-03-22 13:03    [W:0.031 / U:0.480 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site