Messages in this thread | | | From | Chris Mason <> | Subject | Re: 2.6.12.2 dies after 24 hours | Date | Tue, 12 Jul 2005 20:42:09 -0400 |
| |
On Tuesday 12 July 2005 20:27, Rob Mueller wrote: > > > We're also applying the attached patch. There's a bug in reiserfs that > > > gets tickled by our huge MMAP usage (it's amazing what really busy > > > Cyrus daemons can do to a server, ouch). It's fixed in generic_write, > > > so we take the few percent performance hit for something that doesn't > > > break! > > > > Interesting - When I got the problem it was on mail servers under high > > load (handling 60.000 emails pr. hour) with reiserfs as file system. I > > have seen this problem on 5 different servers so I am confident that > > it is not hardware failure. > > > > Sometimes the server load just rises and then the server dies other > > times the load rises but the kernel manages to get it back alive > > filling up syslog with messages like this > > Sounds like a different issue. The patch Bron included before fixes (or at > least reduces to the point where it fixes it for us) a problem where > processes get stuck in D state and are unkillable. A reboot is required to > remove them. Apparently this is a known bug in ReiserFS (see messages > below). As noted, the same bug exists in ext3. There appears to have been > some patches to try and fix it for both reiserfs and ext3, but I'm not sure > if they're in the mainline kernel yet. > > http://www.ussg.iu.edu/hypermail/linux/kernel/0409.0/2056.html > http://hulllug.principalhosting.net/archive/index.php/t-22774.html >
There is a much less complex solution that I've just recently gotten working in the SUSE kernel. If reiser3/ext3 don't log the inode during atime updates, the problem goes away.
You can solve this now by mounting with -o noatime (although that might not play well with cyrus, not sure). My current patch works around this in ugly ways, what I plan on doing during OLS is finding out why ext3 is still logging the inode all the time.
For reiser3, this was to avoid kswapd having to log a bunch of inodes in response to memory pressure, but that was back in 2.4 when things were different. We shouldn't need to do it anymore...
-chris - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |