lkml.org 
[lkml]   [2004]   [Feb]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: 2.6.1 IO lockup on SMP systems


On Mon, 23 Feb 2004, Andrew Morton wrote:

> "Sergey S. Kostyliov" <rathamahata@php4.ru> wrote:
> >
> > > > OK, so everything is stuck trying to allocate memory. Perhaps you ran out
> > > > of swapspace, or some process has gone berzerk allocating memory.
> >
> > The memory exhaustion is indeed possible for this box. I'll double check
> > ulimit and /etc/security/limits.conf stuff. The only thing which worries
> > me that this box had been running for months without any problems with
> > 2.4.23aa1.
>
> It is conceivable that you have some application which runs OK on 2.4.x but
> has some subtle bug which causes the app to go crazy on a 2.6 kernel
> consuming lots of memory. Or there's a bug in the 2.6 kernel ;)
>
> > I have added another 2Gb to swap space (hope this give enough time
> > to find the memory hungry process(es)).
> >
> > > >
> > > > How much memory does the machine have, and how much swap space?
> > > >
> > > # free
> > > total used free shared buffers cached
> > > Mem: 2073868 2067508 6360 0 232708 897828
> > > -/+ buffers/cache: 936972 1136896
> > > Swap: 1535976 5228 1530748
> > >
> > > > I suggest that you run a `vmstat 30' trace on a terminal somewhere, see
> > > > what it says prior to the hangs.
> > > Ok.We'll try to get it next time.
> >
> > Here it is:
> > procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
> > r b swpd free buff cache si so bi bo in cs us sy id wa
> > 1 0 551920 8108 203744 933532 0 0 4 68 1214 426 5 1 92 2
> > 0 0 551928 7140 203756 930316 0 0 17 61 1240 529 8 1 89 2
> > 0 0 551976 5788 203772 928224 1 6 360 139 1297 317 7 2 83 8
> > 0 0 551968 7588 203812 923504 0 0 19 125 1303 308 8 2 87 4
> > 0 1 551976 10444 203892 914100 0 0 25 127 1433 438 10 3 85 3
> > 0 0 551976 9220 204004 914804 0 0 123 126 1278 325 6 1 88 5
> > 0 0 551976 8108 204044 912248 0 0 38 69 1279 291 6 1 91 2
> > 0 1 551976 11828 204144 912320 1 0 135 94 1249 296 6 1 89 3
> > 0 5 562204 3280 203952 157084 1 566 305 674 1281 313 6 4 73 17
> > 0 18 598224 4276 1888 33356 91 2734 233 2761 1090 199 0 2 0 97
> > 1 38 662520 2760 2104 30520 110 3721 261 3738 1161 831 1 2 0 97
> > 10 41 699936 2772 1920 28716 123 2924 249 2946 1103 1273 0 3 0 97
> > 0 39 748588 2956 1956 22668 160 3313 245 3331 1056 1047 0 2 0 98
> > 0 38 796100 3108 1888 21348 321 3191 430 3206 1045 1002 0 2 0 97
> > 4 43 844532 3308 1956 17644 518 3719 670 3733 1357 999 0 2 0 98
> > 0 51 882596 2940 2052 13960 520 2796 705 2810 1048 1182 0 2 0 98
> > 3 59 913392 2456 2048 10900 1013 2524 1308 2542 1144 601 0 2 0 98
> > 5 71 937816 2760 2072 8584 1534 2681 1860 2702 1234 607 0 2 0 97
>
> OK, so it's doing a lot of swapping and your swap utilisation is
> continuously increasing. I would suspect an application or kernel memory
> leak.
>
> I suggest you keep that `vmstat 30' running all the time. When the machine
> dies, take a look at the final 20 lines.
>
> Also, run
>
> while true
> do
> cat /proc/meminfo
> sleep 10
> done
>
> and record the info which that leaves behind when the machine locks up.
> This should tell us whether it is an application or kernel memory leak. If
> it is indeed a leak.

Hi Andrew,

Care to explain me why should the kernel hang if due to an application
leak ?

The hang looks wrong even if the leak is in userspace app, yes?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 14:01    [from the cache]
©2003-2011 Jasper Spaans