Messages in this thread Patch in this message | | | Subject | Re: [BUG] kernel freezes with latest tree | From | Peter Zijlstra <> | Date | Wed, 11 Jan 2012 13:25:11 +0100 |
| |
On Wed, 2012-01-11 at 10:04 +0100, Peter Zijlstra wrote: > Maybe adding a few more NEED_BREAK bits > and making it a counter and overflowing it into ABORT might be good. > > I could reproduce and confirm something like the below makes the hang go-away. I haven't managed to fully understand why we're stuck though because we do release the runqueue locks and re-enable IRQs on this lock-break.
My stuck machine had several CPUs stuck in a load-balance pass, so it could be they're bouncing tasks back and forth without actually making any progress what so ever.
I reproduced with hackbench 500, which results in 20000 tasks, spread over 24 cpus that gives some 833 tasks per runqueue on average, easily overflowing that lock-break scanning limit.
--- Subject: sched: Limit load-balance retries on lock-break From: Peter Zijlstra <a.p.zijlstra@chello.nl> Date: Wed Jan 11 13:11:12 CET 2012
Eric and David reported dead machines and traced it to commit a195f004 ("sched: Fix load-balance lock-breaking"), it turns out there's still a scenario where we can end up re-trying forever.
Limit the number of retries and simply abort.
Reported-by: Eric Dumazet <eric.dumazet@gmail.com> Reported-by: David Ahern <dsahern@gmail.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> --- kernel/sched/fair.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
--- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3130,8 +3130,10 @@ task_hot(struct task_struct *p, u64 now, } #define LBF_ALL_PINNED 0x01 -#define LBF_NEED_BREAK 0x02 -#define LBF_ABORT 0x04 +#define LBF_NEED_BREAK 0x02 /* clears into HAD_BREAK */ +#define LBF_HAD_BREAK 0x04 +#define LBF_HAD_BREAKS 0x0C /* count HAD_BREAKs overflows into ABORT */ +#define LBF_ABORT 0x10 /* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu? @@ -4509,6 +4511,9 @@ static int load_balance(int this_cpu, st if (lb_flags & LBF_NEED_BREAK) { lb_flags &= ~LBF_NEED_BREAK; + lb_flags += LBF_HAD_BREAK; + if (lb_flags & LBF_ABORT) + goto out_balanced; goto redo; }
| |