lkml.org 
[lkml]   [2019]   [Aug]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: sched: Unexpected reschedule of offline CPU#2!
On Fri, Aug 16, 2019 at 12:22:22PM +0200, Thomas Gleixner wrote:
> On Mon, 29 Jul 2019, Guenter Roeck wrote:
> > On Mon, Jul 29, 2019 at 12:47:45PM +0200, Peter Zijlstra wrote:
> > > On Mon, Jul 29, 2019 at 12:38:30PM +0200, Thomas Gleixner wrote:
> > > > Reboot has two modes:
> > > >
> > > > - Regular reboot initiated from user space
> > > >
> > > > - Panic reboot
> > > >
> > > > For the regular reboot we can make it go through proper hotplug,
> > >
> > > That seems sensible.
> > >
> > > > for the panic case not so much.
> > >
> > > It's panic, shit has already hit fan, one or two more pieces shouldn't
> > > something anybody cares about.
> > >
> >
> > Some more digging shows that this happens a lot with Google GCE intances,
> > typically after a panic. The problem with that, if I understand correctly,
> > is that it may prevent coredumps from being written. So, while of course
> > the panic is what needs to be fixed, it is still quite annoying, and it
> > would help if this can be fixed for panic handling as well.
> >
> > How about the patch suggested by Hillf Danton ? Would that help for the
> > panic case ?
>
> I have no idea how that patch looks like, but the quick hack is below.
>
> Thanks,
>
> tglx
>
> 8<---------------
> diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> index 75fea0d48c0e..625627b1457c 100644
> --- a/arch/x86/kernel/process.c
> +++ b/arch/x86/kernel/process.c
> @@ -601,6 +601,7 @@ void stop_this_cpu(void *dummy)
> /*
> * Remove this CPU:
> */
> + set_cpu_active(smp_processor_id(), false);
> set_cpu_online(smp_processor_id(), false);
> disable_local_APIC();
> mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
>
No luck. The problem is still seen with this patch applied on top of
the mainline kernel (commit a69e90512d9def6).

Guenter

---
[ 22.315834] e1000e: EEE TX LPI TIMER: 00000000
[ 22.323624] reboot: Restarting system
[ 22.324260] reboot: machine restart
[ 22.325885] ------------[ cut here ]------------
[ 22.330425] sched: Unexpected reschedule of offline CPU#3!
ILLOPC: ffffffffb524403f: 0f 0b
[ 22.330926] WARNING: CPU: 1 PID: 0 at arch/x86/kernel/smp.c:126 native_smp_send_reschedule+0x2f/0x40
[ 22.331238] Modules linked in:
[ 22.331427] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.3.0-rc4+ #1
[ 22.331626] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
[ 22.331971] RIP: 0010:native_smp_send_reschedule+0x2f/0x40
[ 22.332164] Code: 05 de 81 95 01 73 15 48 8b 05 bd fa 61 01 be fd 00 00 00 48 8b 40 30 e9 6f d0 fb 00 89 fe 48 c7 c7 88 da 74 b6 e8 7f 6c 02 00 <0f> 0b c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 55 53 48 83 ec
[ 22.332705] RSP: 0018:ffffa457800d0d68 EFLAGS: 00000086
[ 22.332884] RAX: 0000000000000000 RBX: ffff9a8cbb9ba000 RCX: 0000000000000103
[ 22.333109] RDX: 0000000080000103 RSI: 0000000000000000 RDI: 00000000ffffffff
[ 22.333327] RBP: ffffa457800d0e90 R08: 0000000000000000 R09: 0000000000000000
[ 22.333546] R10: 0000000000000000 R11: ffffa457800d0c10 R12: 000000000000a1b9
[ 22.333767] R13: ffff9a8cbae26030 R14: ffff9a8cbae25f80 R15: ffff9a8cbb83a000
[ 22.334045] FS: 0000000000000000(0000) GS:ffff9a8cbb880000(0000) knlGS:0000000000000000
[ 22.334321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 22.334520] CR2: 00007fba66a35010 CR3: 0000000176cd6000 CR4: 00000000007406e0
[ 22.334794] PKRU: 55555554
[ 22.334915] Call Trace:
[ 22.335062] <IRQ>
[ 22.335148] check_preempt_curr+0x7f/0xc0
[ 22.335295] load_balance+0x589/0xc50
[ 22.335513] rebalance_domains+0x30d/0x410
[ 22.335684] _nohz_idle_balance+0x1bd/0x200
[ 22.335854] __do_softirq+0xe5/0x478
[ 22.336023] irq_exit+0xa9/0xc0
[ 22.336163] reschedule_interrupt+0xf/0x20
[ 22.336317] </IRQ>
[ 22.336409] RIP: 0010:default_idle+0x23/0x180
[ 22.336561] Code: ff 90 90 90 90 90 90 41 55 41 54 55 53 e8 45 75 7c ff 0f 1f 44 00 00 e8 0b aa 40 ff e9 07 00 00 00 0f 00 2d 31 94 4a 00 fb f4 <e8> 28 75 7c ff 89 c5 0f 1f 44 00 00 5b 5d 41 5c 41 5d c3 65 8b 05
[ 22.337102] RSP: 0018:ffffa4578006bec0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff02
[ 22.337342] RAX: ffff9a8cbae23fc0 RBX: 0000000000000001 RCX: 0000000000000001
[ 22.337561] RDX: 0000000000000046 RSI: 0000000000000006 RDI: ffffffffb6852dd6
[ 22.337780] RBP: ffffffffb6b9c1f8 R08: 0000000000000001 R09: 0000000000000000
[ 22.337996] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 22.338229] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 22.338501] do_idle+0x1df/0x260
[ 22.338588] ? _raw_spin_unlock_irqrestore+0x4c/0x60
[ 22.338706] cpu_startup_entry+0x14/0x20
[ 22.338793] start_secondary+0x151/0x180
[ 22.338885] secondary_startup_64+0xa4/0xb0
[ 22.339060] irq event stamp: 61631
[ 22.339176] hardirqs last enabled at (61630): [<ffffffffb5f5c6dc>] _raw_spin_unlock_irqrestore+0x4c/0x60
[ 22.339373] hardirqs last disabled at (61631): [<ffffffffb5f5c46d>] _raw_spin_lock_irqsave+0xd/0x50
[ 22.339568] softirqs last enabled at (61626): [<ffffffffb5272bc8>] irq_enter+0x58/0x60
[ 22.339726] softirqs last disabled at (61627): [<ffffffffb5272c79>] irq_exit+0xa9/0xc0
[ 22.339897] ---[ end trace 8ad53445879058cc ]---
[ 22.340384] ACPI MEMORY or I/O RESET_REG.

\
 
 \ /
  Last update: 2019-08-16 21:33    [W:0.081 / U:0.772 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site