Messages in this thread | | | Date | Tue, 9 Jun 2009 01:30:06 +0200 | From | Johannes Weiner <> | Subject | Re: Probablem with dropping caches |
| |
[adding Ccs]
On Thu, Jun 04, 2009 at 12:21:22AM -0600, Bruce Guenter wrote: > Hello. > > I am having a problem with a system that appears to be spontaneously > dropping large parts of its caches. The work load on this system is > primarily I/O bound (it's a mailbox server), and as such the loss of > cache memory is causing severe performance degradation. > > For example, here is some output from vmstat 1: > > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 1 1 0 87164 683212 1069748 0 0 544 164 856 730 3 2 91 4 > 0 1 0 81124 689100 1069508 0 0 5880 104 1070 834 2 1 74 23 > 0 1 0 89956 691588 1057408 0 0 5288 0 1163 915 0 2 72 25 > 0 1 0 138020 690652 1012444 0 0 5724 0 1136 831 2 0 75 23 > 0 0 0 243384 690460 906500 0 0 4716 0 1282 844 0 2 61 36 > 0 1 0 294704 690152 854232 0 0 1108 428 1123 1093 2 2 81 15 > 0 0 0 285984 690380 854504 0 0 252 0 721 671 3 1 92 3 > 0 1 0 426844 690780 722408 0 0 3096 1748 1197 846 1 2 84 13 > 0 1 0 579684 691232 568344 0 0 4228 156 1300 1083 2 2 69 27 > 1 1 0 676312 691832 467244 0 0 5256 0 1072 741 0 2 75 23 > > As far as I can tell from df and similar reporting, there are not > hundreds of MB of files being deleted, which would have similar > behavior. It is not swapping, nor is memory actually leaking (since > free memory + cache is nearly constant). All of the active programs run > with small memory ulimits and as such are not consuming and then > releasing hundreds of MB of memory. > > There are also intervals where the system is reading several MB per > second but the caches do not grow significantly: > > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 0 35 0 960396 749544 62416 0 0 7396 0 1424 1154 2 2 0 96 > 1 34 0 963868 750536 62252 0 0 8596 208 1695 1463 4 3 0 93 > 0 38 0 967452 752800 62980 0 0 7176 20 1378 972 4 1 0 95 > 0 38 0 968100 751308 61400 0 0 7260 180 1423 1109 3 2 0 95 > 2 42 0 966252 751540 61872 0 0 8196 0 1404 1328 1 1 0 97 > 0 43 0 955440 751956 60520 0 0 8692 0 1846 1925 5 3 0 92 > 2 49 0 943644 752836 61412 0 0 9324 200 1783 1582 5 3 0 92 > 1 39 0 959368 751892 62104 0 0 7836 64 1874 1855 9 5 0 86 > > This system has 2GB RAM and 4 72GB drives in a 3Ware RAID10 array. The > active filesystem is ext4 with the following mount options: > > noatime,nodiratime,data=journal > > The data=journal option comes from benchmarking I did a while back that > indicated it was best for sync+unlink heavy work loads such as this one > has. I have remounted with data=ordered but that did not solve the > problem. > > The kernel (as of now) is 2.6.29.4 compiled with gcc 3.4.6 on Gentoo. > > I also have another system, which is similarly configured but is using > the ext3 filesystem. It does not exhibit this behavior which leads me > to suspect some difference between ext3 and ext4 is causing the problem. > I however have no other evidence to point a finger at ext4, and am at a > loss as to what else to investigate. > > Has anybody else seen this behavior before? What other details can I > investigate to figure out what is causing this problem? What other information > would be useful to diagnose this? > > Thank you. > > -- > Bruce Guenter <bruce@untroubled.org> http://untroubled.org/
| |