[lkml]   [1996]   [Oct]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subjectsyslog spiral death
Hi all,

Have been seeing a problem with a system we have that has quite a
heavy load - lots of short lived processes (upto 3 per second which can
run for a few sconds to a few minutes).

The problem is, that each process adds logs to syslog (for
accounting). Everything usually runs fine, but it appears that when
the syslog file gets large (say 8G) that syslog starts using more of
the CPU, which creates a backlog in the processes, which results in a
higher load, which means more syslog acitivity etc.

When the syslog file is small, the system has no problems recovering
from bursts of high load (loadavg of over 50). But when the syslog
file is large, things just compound, and the loadavg goes through the
ceiling (over 400) and things are backed up until the system cannot
create anymore processes.

The processes are basically transfering small files across the network.

Anyone know what causes this behaviour? Inefficiency in the
filesystem implementation (ext2) for large files?



Tony Gale

| All opinions expressed are my own |

 \ /
  Last update: 2005-03-22 13:38    [W:0.105 / U:0.124 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site