lkml.org 
[lkml]   [2011]   [May]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    Date
    SubjectRe: Kernel falls apart under light memory pressure (i.e. linking vmlinux)
    From
    On Thu, May 19, 2011 at 11:15 AM, Andrew Lutomirski <luto@mit.edu> wrote:
    > On Wed, May 18, 2011 at 1:17 AM, Minchan Kim <minchan.kim@gmail.com> wrote:
    >> On Wed, May 18, 2011 at 4:22 AM, Andrew Lutomirski <luto@mit.edu> wrote:
    >>>> No, thanks. However it would be valuable if you can retry with this
    >>>> patch _alone_ (without the "if (need_resched()) return false;" change,
    >>>> as I don't see how it helps your case).
    >>>>
    >>>> @@ -2286,7 +2290,7 @@ static bool sleeping_prematurely(pg_data_t
    >>>> *pgdat, int order, long remaining,
    >>>>        * must be balanced
    >>>>        */
    >>>>       if (order)
    >>>> -               return pgdat_balanced(pgdat, balanced, classzone_idx);
    >>>> +               return !pgdat_balanced(pgdat, balanced, classzone_idx);
    >>>>       else
    >>>>               return !all_zones_ok;
    >>>>  }
    >>>
    >>> Done.
    >>>
    >>> I logged in, added swap, and ran a program that allocated 1900MB of
    >>> RAM and memset it.  The system lagged a bit but survived.  kswapd
    >>> showed 10% CPU (which is odd, IMO, since I'm using aesni-intel and I
    >>> think that all the crypt happens in kworker when aesni-intel is in
    >>> use).
    >>
    >> I think kswapd could use 10% enough for reclaim.
    >>
    >>>
    >>> Then I started Firefox, loaded gmail, and ran test_mempressure.sh.
    >>> Kaboom!  (I.e. system was hung)  SysRq-F saved the system and produced
    >>
    >> Hang?
    >> It means you see softhangup of kswapd? or mouse/keyboard doesn't move?
    >
    > Mouse and keyboard dead.
    >
    >> Andrew, Could you test this patch with !pgdat_balanced patch?
    >> I think we shouldn't see OOM message if we have lots of free swap space.
    >>
    >> == CUT_HERE ==
    >> diff --git a/mm/vmscan.c b/mm/vmscan.c
    >> index f73b865..cc23f04 100644
    >> --- a/mm/vmscan.c
    >> +++ b/mm/vmscan.c
    >> @@ -1341,10 +1341,6 @@ static inline bool
    >> should_reclaim_stall(unsigned long nr_taken,
    >>        if (current_is_kswapd())
    >>                return false;
    >>
    >> -       /* Only stall on lumpy reclaim */
    >> -       if (sc->reclaim_mode & RECLAIM_MODE_SINGLE)
    >> -               return false;
    >> -
    >>        /* If we have relaimed everything on the isolated list, no stall */
    >>        if (nr_freed == nr_taken)
    >>                return false;
    >>
    >>
    >>
    >> Then, if you don't see any unnecessary OOM but still see the hangup,
    >> could you apply this patch based on previous?
    >
    > With this patch, I started GNOME and Firefox, turned on swap, and ran
    > test_mempressure.sh 1500 1400 1.  Instant panic (or OOPS and hang or
    > something -- didn't get the top part).  Picture attached -- it looks
    > like memcg might be involved.  I'm running F15, so it might even be
    > doing something.

    I cannot figure out why happens OOPS.
    Let me know your kernel version and config.
    Kame. Is there anything related to memcg you guess?

    In addition, the patch I give was utterly stupid.
    The goal is that we wait dirty page writeback in (order-0 | high
    priority) reclaim.
    (But I don't think it's ideal solution in this problem but just for
    proving the problem)
    But although we pass sync with 1 in set_reclaim_mode, it ignores.
    So fix is following as. (NOTICE: It doesn't related to your OOPS. )
    But before further experiment, let's fix your oops.

    diff --git a/mm/vmscan.c b/mm/vmscan.c
    index 292582c..69d317e 100644
    --- a/mm/vmscan.c
    +++ b/mm/vmscan.c
    @@ -311,7 +311,8 @@ static void set_reclaim_mode(int priority, struct
    scan_control *sc,
    */
    if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
    sc->reclaim_mode |= syncmode;
    - else if (sc->order && priority < DEF_PRIORITY - 2)
    + else if ((sc->order && priority < DEF_PRIORITY - 2) ||
    + prioiry <= DEF_PRIORITY / 3)
    sc->reclaim_mode |= syncmode;
    else
    sc->reclaim_mode = RECLAIM_MODE_SINGLE | RECLAIM_MODE_ASYNC;
    @@ -1349,10 +1350,6 @@ static inline bool
    should_reclaim_stall(unsigned long nr_taken,
    if (current_is_kswapd())
    return false;

    - /* Only stall on lumpy reclaim */
    - if (sc->reclaim_mode & RECLAIM_MODE_SINGLE)
    - return false;
    -
    /* If we have relaimed everything on the isolated list, no stall */
    if (nr_freed == nr_taken)
    return false;


    >
    > I won't be able to get netconsole dumps until next week because I'm
    > out of town and only have this one computer here.

    No problem. :)
    We should avoid OOPS for the experiment.


    >
    > I haven't tried the other patch.
    >
    > Also, the !pgdat_balanced fix plus the if (need_resched()) return
    > false patch just hung once on 2.6.37-rc9.  I don't know what triggered

    Thanks for the good information.
    It seems need_resched patch isn't good candidate to fix current problem.
    We already weeded it out.

    Thank you very much for the testing!

    > it.  Maybe yum.
    >
    > --Andy
    >



    --
    Kind regards,
    Minchan Kim
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2011-05-19 04:57    [W:0.032 / U:212.984 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site