lkml.org 
[lkml]   [2012]   [Jan]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: ZenIV (was: Re: Status of arm-soc.git for 3.2)
On Wed, 4 Jan 2012, Russell King - ARM Linux wrote:

> On Wed, Jan 04, 2012 at 07:10:42PM +0000, Russell King - ARM Linux wrote:
> > On Wed, Jan 04, 2012 at 01:55:48PM -0500, Nicolas Pitre wrote:
> > > On Wed, 4 Jan 2012, Russell King - ARM Linux wrote:
> > > > I've now disabled all access to my git tree there, and restarted apache.
> > > > We will see whether that improves stability - I suspect it will do because
> > > > I reckon that the problem is that the smart git stuff is what's killing
> > > > the machine.
> > >
> > > I'm sure that is the case. However faulty hardware could still be the
> > > root cause, but without the load from Git the machine might become
> > > loaded lightly enough you might not see any ill effects before quite a
> > > while.
> >
> > When it's serving the next Fedora release (it's one of the official
> > mirrors) I'm sure any problems like that would become noticable - but
> > they haven't yet. It purely seems to be something git is doing which
> > is killing the machine.
>
> I think we've just found the issue causing httpd to die:
>
> - Fedora systems set a limit at login time on the number of processes a
> user can run - which is set to a soft limit of 1024.
>
> - When someone logs in, their shell inherits this soft rlimit. This
> gets inherited by all sub-processes, including through a su to their
> root shell.
>
> - When they restart httpd, httpd also inherits this limit.
>
> - httpd's own internal limits are set to a max clients of 1024.
>
> The problem comes when httpd hits 1024 processes - as it forks as root,
> this succeeds (root is not subjected to this rlimit), and then a
> subsequent setuid() fails with -EAGAIN, causing httpd to experience a
> fatal error.
>
> Obviously, this is not a good combination of things to happen. It's
> also completely unnoticable to anyone who restarts apache.

Nasty.

> So, I've 'fixed' it by raising the rlimit in the httpd startup scripts,
> which should keep it fixed whenever anyone else restarts httpd.
>
> The problem should be solved, and as such I've re-enabled access to
> the git tree.

OK, let's hope things will hold up.

I suspect this might not be correlated to the memory exhaustion ZenIV
experienced in the past though. That might require another kind of
solution if that comes again.


Nicolas


\
 
 \ /
  Last update: 2012-01-04 21:35    [from the cache]
©2003-2011 Jasper Spaans