lkml.org 
[lkml]   [2012]   [Jan]   [11]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
SubjectRe: [BUG] kernel freezes with latest tree
From
Date
On Wed, 2012-01-11 at 10:04 +0100, Peter Zijlstra wrote:
> Maybe adding a few more NEED_BREAK bits
> and making it a counter and overflowing it into ABORT might be good.
>
>
I could reproduce and confirm something like the below makes the hang
go-away. I haven't managed to fully understand why we're stuck though
because we do release the runqueue locks and re-enable IRQs on this
lock-break.

My stuck machine had several CPUs stuck in a load-balance pass, so it
could be they're bouncing tasks back and forth without actually making
any progress what so ever.

I reproduced with hackbench 500, which results in 20000 tasks, spread
over 24 cpus that gives some 833 tasks per runqueue on average, easily
overflowing that lock-break scanning limit.

---
Subject: sched: Limit load-balance retries on lock-break
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date: Wed Jan 11 13:11:12 CET 2012

Eric and David reported dead machines and traced it to commit a195f004 ("sched:
Fix load-balance lock-breaking"), it turns out there's still a
scenario where we can end up re-trying forever.

Limit the number of retries and simply abort.

Reported-by: Eric Dumazet <eric.dumazet@gmail.com>
Reported-by: David Ahern <dsahern@gmail.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
kernel/sched/fair.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3130,8 +3130,10 @@ task_hot(struct task_struct *p, u64 now,
}

#define LBF_ALL_PINNED 0x01
-#define LBF_NEED_BREAK 0x02
-#define LBF_ABORT 0x04
+#define LBF_NEED_BREAK 0x02 /* clears into HAD_BREAK */
+#define LBF_HAD_BREAK 0x04
+#define LBF_HAD_BREAKS 0x0C /* count HAD_BREAKs overflows into ABORT */
+#define LBF_ABORT 0x10

/*
* can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
@@ -4509,6 +4511,9 @@ static int load_balance(int this_cpu, st

if (lb_flags & LBF_NEED_BREAK) {
lb_flags &= ~LBF_NEED_BREAK;
+ lb_flags += LBF_HAD_BREAK;
+ if (lb_flags & LBF_ABORT)
+ goto out_balanced;
goto redo;
}



\
 
 \ /
  Last update: 2012-01-11 13:27    [W:0.400 / U:0.416 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site