Messages in this thread | | | From | Ani Sinha <> | Date | Mon, 14 Dec 2015 21:54:41 +0530 | Subject | Re: new warning on sysrq kernel crash trigger |
| |
Rik, any comments?
On Sat, Dec 12, 2015 at 6:33 AM, Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote: > On Fri, Dec 11, 2015 at 04:16:37PM -0800, Ani Sinha wrote: >> On Fri, 11 Dec 2015, Paul E. McKenney wrote: >> > On Fri, Dec 11, 2015 at 05:10:43PM -0500, Rik van Riel wrote: >> > > On 12/11/2015 03:44 PM, Ani Sinha wrote: >> > > > >> > > > >> > > > On Thu, 10 Dec 2015, Paul E. McKenney wrote: >> > > > >> > > >> On Thu, Dec 10, 2015 at 03:57:09PM -0800, Ani Sinha wrote: >> > > >>> Hi guys >> > > >>> >> > > >>> I am noticing a new warning in linux 3.18 which we did not see before >> > > >>> in linux 3.4 : >> > > >>> >> > > >>> bash-4.1# echo c > /proc/sysrq-trigger >> > > >>> [ 978.807185] BUG: sleeping function called from invalid context at >> > > >>> ../arch/x86/mm/fault.c:1187 >> > > >>> [ 978.909816] in_atomic(): 0, irqs_disabled(): 0, pid: 4706, name: bash >> > > >>> [ 978.987358] Preemption disabled at:[<ffffffff81484339>] printk+0x48/0x4a >> > > >>> >> > > >>> >> > > >>> I have bisected this to the following change : >> > > >>> >> > > >>> commit 984d74a72076a12b400339973e8c98fd2fcd90e5 >> > > >>> Author: Rik van Riel <riel@redhat.com> >> > > >>> Date: Fri Jun 6 14:38:13 2014 -0700 >> > > >>> >> > > >>> sysrq: rcu-ify __handle_sysrq >> > > >>> >> > > >>> >> > > >>> the rcu_read_lock() in handle_sysrq() bumps up >> > > >>> current->rcu_read_lock_nesting. Hence, in __do_page_fault() when it >> > > >>> calls might_sleep() in x86/mm/fault.c line 1191, >> > > >>> preempt_count_equals(0) returns false and hence the warning is >> > > >>> printed. >> > > >>> >> > > >>> One way to handle this would be to do something like this: >> > > >>> >> > > >>> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c >> > > >>> index eef44d9..d4dbe22 100644 >> > > >>> --- a/arch/x86/mm/fault.c >> > > >>> +++ b/arch/x86/mm/fault.c >> > > >>> @@ -1132,7 +1132,7 @@ __do_page_fault(struct pt_regs *regs, unsigned >> > > >>> long error_code, >> > > >>> * If we're in an interrupt, have no user context or are running >> > > >>> * in a region with pagefaults disabled then we must not take the fault >> > > >>> */ >> > > >>> - if (unlikely(faulthandler_disabled() || !mm)) { >> > > >>> + if (unlikely(faulthandler_disabled() || rcu_preempt_depth() || !mm)) { >> > > >> >> > > >> This works if CONFIG_PREEMPT=y, but if CONFIG_PREEMPT=n, then >> > > >> rcu_preempt_depth() unconditionally returns zero. And if >> > > >> CONFIG_PREEMPT_COUNT=y && CONFIG_PREEMPT=n, you would still see >> > > >> the might_sleep() splat. >> > > >> >> > > >> Maybe use SRCU instead of RCU for this purpose? >> > > >> >> > > > >> > > > From ae232ce3fb167b2ad363bfac7aab69001bc55a50 Mon Sep 17 00:00:00 2001 >> > > > From: Ani Sinha <ani@arista.com> >> > > > Date: Fri, 11 Dec 2015 12:07:42 -0800 >> > > > Subject: [PATCH 1/1] Fix 'sleeping function called from invalid context' >> > > > warning in sysrq generated crash. >> > > > >> > > > Commit 984d74a72076a1 ("sysrq: rcu-ify __handle_sysrq") >> > > > replaced spin_lock_irqsave() calls with >> > > > rcu_read_lock() calls in sysrq. Since rcu_read_lock() does not >> > > > disable preemption, faulthandler_disabled() in >> > > > __do_page_fault() in x86/fault.c returns false. When the code >> > > > later calls might_sleep() in the pagefault handler, we get the >> > > > following warning: >> > > > >> > > > BUG: sleeping function called from invalid context at ../arch/x86/mm/fault.c:1187 >> > > > in_atomic(): 0, irqs_disabled(): 0, pid: 4706, name: bash >> > > > Preemption disabled at:[<ffffffff81484339>] printk+0x48/0x4a >> > > > >> > > > To fix this, replace RCU call in handle_sysrq() to use SRCU. >> > > >> > > The sysrq code can be called from irq context. >> > > >> > > Trying to use SRCU from an irq context sounds like it could >> > > be a bad idea, though admittedly I do not know enough about >> > > SRCU to know for sure :) >> > >> > Indeed, not the best idea! ;-) >> > >> > I could imagine something like this: >> > >> > if (in_irq()) >> > rcu_read_lock(); >> > else >> > idx = srcu_read_lock(&sysrq_rcu); >> > >> > And ditto for unlock. Then, for the update: >> > >> > synchronize_rcu_mult(call_rcu, call_sysrq_srcu); >> > >> > Where: >> > >> > static void call_sysrq_srcu(struct rcu_head *head, rcu_callback_t func) >> > { >> > call_srcu(&sysrq_rcu, head, func); >> > } >> > >> >> >From ae232ce3fb167b2ad363bfac7aab69001bc55a50 Mon Sep 17 00:00:00 2001 >> From: Ani Sinha <ani@arista.com> >> Date: Fri, 11 Dec 2015 12:07:42 -0800 >> Subject: [PATCH 1/1] Fix 'sleeping function called from invalid context' >> warning in sysrq generated crash. >> >> Commit 984d74a72076a1 ("sysrq: rcu-ify __handle_sysrq") >> replaced spin_lock_irqsave() calls with >> rcu_read_lock() calls in sysrq. Since rcu_read_lock() does not >> disable preemption, faulthandler_disabled() in >> __do_page_fault() in x86/fault.c returns false. When the code >> later calls might_sleep() in the pagefault handler, we get the >> following warning: >> >> BUG: sleeping function called from invalid context at ../arch/x86/mm/fault.c:1187 >> in_atomic(): 0, irqs_disabled(): 0, pid: 4706, name: bash >> Preemption disabled at:[<ffffffff81484339>] printk+0x48/0x4a >> >> To fix this, replace RCU call in handle_sysrq() to use SRCU >> in non-irq context. >> >> Tested this patch on linux 3.18 by booting off one of our boards. >> >> Fixes: 984d74a72076a1 ("sysrq: rcu-ify __handle_sysrq") >> >> Signed-off-by: Ani Sinha <ani@arista.com> > > From an RCU perspective: > > Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> > > But I must defer to Rik from an sysrq perspective. > > Thanx, Paul > >> --- >> diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c >> index 5381a72..df7d747 100644 >> --- a/drivers/tty/sysrq.c >> +++ b/drivers/tty/sysrq.c >> @@ -54,6 +54,7 @@ >> /* Whether we react on sysrq keys or just ignore them */ >> static int __read_mostly sysrq_enabled = CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE; >> static bool __read_mostly sysrq_always_enabled; >> +DEFINE_STATIC_SRCU(sysrq_rcu); >> >> static bool sysrq_on(void) >> { >> @@ -519,10 +520,13 @@ void __handle_sysrq(int key, bool check_mask) >> { >> struct sysrq_key_op *op_p; >> int orig_log_level; >> - int i; >> + int i, idx; >> >> rcu_sysrq_start(); >> - rcu_read_lock(); >> + if (in_irq()) >> + rcu_read_lock(); >> + else >> + idx = srcu_read_lock(&sysrq_rcu); >> /* >> * Raise the apparent loglevel to maximum so that the sysrq header >> * is shown to provide the user with positive feedback. We do not >> @@ -564,7 +568,10 @@ void __handle_sysrq(int key, bool check_mask) >> pr_cont("\n"); >> console_loglevel = orig_log_level; >> } >> - rcu_read_unlock(); >> + if (in_irq()) >> + rcu_read_unlock(); >> + else >> + srcu_read_unlock(&sysrq_rcu, idx); >> rcu_sysrq_end(); >> } >> >> @@ -1040,6 +1047,11 @@ int sysrq_toggle_support(int enable_mask) >> return 0; >> } >> >> +static void call_sysrq_srcu(struct rcu_head *head, rcu_callback_t func) >> +{ >> + call_srcu(&sysrq_rcu, head, func); >> +} >> + >> static int __sysrq_swap_key_ops(int key, struct sysrq_key_op *insert_op_p, >> struct sysrq_key_op *remove_op_p) >> { >> @@ -1059,7 +1071,7 @@ static int __sysrq_swap_key_ops(int key, struct sysrq_key_op *insert_op_p, >> * Wait for it to go away before returning, so the code for an old >> * op is not freed (eg. on module unload) while it is in use. >> */ >> - synchronize_rcu(); >> + synchronize_rcu_mult(call_rcu, call_sysrq_srcu); >> >> return retval; >> } >> >
| |