lkml.org 
[lkml]   [2020]   [Mar]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: INFO: rcu detected stall in sys_keyctl
On Wed, Mar 4, 2020 at 9:36 PM Kris Karas <linux-1993@moonlit-rail.com> wrote:
>
> Dmitry Vyukov wrote:
> > Kris Karas wrote:
> >> [...]
> >> rcu: INFO: rcu_sched self-detected stall on CPU
> >> rcu: 14-....: (20999 ticks this GP)
> >> idle=216/1/0x4000000000000002 softirq=454/454 fqs=5250
> >> (t=21004 jiffies g=-755 q=1327)
> >> NMI backtrace for cpu 14
> >> CPU: 14 PID: 520 Comm: pidof Tainted: G D 5.5.7 #1
> >> [...]
> >> I don't have a reproducer for it, either. It showed up in 5.5.7 (but
> >> might be from earlier as it reproduces so infrequently).
> > Hi Kris,
> >
> > What follows after this stack? That's the most interesting part. The
> > part that you showed is common for all stalls and does not mean much,
> > besides the fact that there is a stall. These can well be very
> > different stalls in different parts of kernel.
>
> Hi Dmitry,
>
> Sorry, dummy me, I should have found my original post in Lore and posted
> a link to that.
> Oh, here we go:
>
> https://lore.kernel.org/lkml/6d4f9ac8-a478-2ae4-0fe3-5d074d267148@moonlit-rail.com/

These all mention ptrace_may_access, so that looks like a different bug to me.

> Given that the stall stack is not terribly useful, it would seem that
> the OOPS I saw was probably unrelated to this one caught by syzbot,
> though the stalled CPU does make me curious (as in all the OOPSen I've
> encountered in the past 25 years have rarely mentioned an RCU stall).

Well, it means you don't stress the kernel hard enough :)
Search for "rcu detected stall" here:
https://syzkaller.appspot.com/upstream
https://syzkaller.appspot.com/upstream/fixed
https://syzkaller.appspot.com/linux-4.19
https://syzkaller.appspot.com/linux-4.14
and these are all different _bugs_, some of them have tens of
thousands of crash instances. And that's just in 2.5 years :)


> For convenience, I'll re-post everything I was able to salvage from
> dmesg originally.
>
> Kris
>
> The OOPS in the dump, below, occurred while the machine was booting,
> right about the time that /sbin/init switched from runstate S => 3.
> System daemons (haveged, named, syslogd, etc...) were starting. The OOPS
> occurred in /bin/pidof, which is no doubt checking whether a daemon is
> up before attempting to start it. Under the OOPS, the filesystem was
> functioning (at least well enough to save dmesg to a file), though many
> things were hanging. It required an Alt-SysRq-E to get me a login
> prompt, and I lucked out in PAM and friends working well enough to give
> me a functioning command prompt. Alt-SysRq-S,U,S,B was necessary to
> reboot. Without further ado...
>
> BUG: kernel NULL pointer dereference, address: 00000000000000e8
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 7f6a50067 P4D 7f6a50067 PUD 7f6a51067 PMD 0
> Oops: 0000 [#1] SMP
> CPU: 3 PID: 516 Comm: pidof Not tainted 5.5.7 #1
> Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi, BIOS P3.50 07/18/2019
> RIP: 0010:cap_capable+0x13/0x70
> Code: bf f4 ff ff ff 66 90 e9 01 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 4c 8b 87 88 00 00 00 4c 39 c6 74 39 45 8b 88 e8 00 00 00 <44> 39 8e e8 00 00 00 7e 18 48 8b 86 e0 00 00 00 4c 39 c0 74 12 48
> RSP: 0018:ffffc90000777cb0 EFLAGS: 00010207
> RAX: ffff8887f96ea000 RBX: 0000000000000002 RCX: 0000000000000002
> RDX: 0000000000000013 RSI: 0000000000000000 RDI: ffff8887f9646480
> RBP: 0000000000000013 R08: ffffffff82423da0 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> R13: ffff8887f9646480 R14: ffffffff822a7620 R15: ffff8887fae600c0
> FS: 00007f8ee26cd740(0000) GS:ffff8887fecc0000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000000e8 CR3: 00000007f8691000 CR4: 00000000003406e0
> Call Trace:
> security_capable+0x36/0x50
> ptrace_has_cap+0x14/0x30
> __ptrace_may_access+0x76/0x110
> ptrace_may_access+0x28/0x50
> do_task_stat+0x7b/0xd90
> ? do_filp_open+0xab/0x100
> proc_single_show+0x54/0xc0
> ? __kmalloc+0x183/0x210
> seq_read+0xbb/0x3c0
> vfs_read+0xc6/0x150
> ksys_read+0x6b/0x100
> do_syscall_64+0x3d/0x120
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7f8ee27d682e
> Code: c0 e9 f6 fe ff ff 50 48 8d 3d b6 5d 0a 00 e8 e9 fd 01 00 66 0f 1f 84 00 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 0f 05 <48> 3d 00 f0 ff ff 77 5a c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 28
> RSP: 002b:00007ffdc7fdcf58 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> RAX: ffffffffffffffda RBX: 00007f8ee28ce958 RCX: 00007f8ee27d682e
> RDX: 0000000000000400 RSI: 00000000017e2590 RDI: 0000000000000004
> RBP: 00007f8ee28ce950 R08: 00007f8ee28ac120 R09: 00007ffdc7fdce00
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
> R13: 0000000000000000 R14: 0000000000000000 R15: 00007ffdc7fddf02
> Modules linked in:
> CR2: 00000000000000e8
> ---[ end trace 9da0e81512fbb929 ]---
> RIP: 0010:cap_capable+0x13/0x70
> Code: bf f4 ff ff ff 66 90 e9 01 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 4c 8b 87 88 00 00 00 4c 39 c6 74 39 45 8b 88 e8 00 00 00 <44> 39 8e e8 00 00 00 7e 18 48 8b 86 e0 00 00 00 4c 39 c0 74 12 48
> RSP: 0018:ffffc90000777cb0 EFLAGS: 00010207
> RAX: ffff8887f96ea000 RBX: 0000000000000002 RCX: 0000000000000002
> RDX: 0000000000000013 RSI: 0000000000000000 RDI: ffff8887f9646480
> RBP: 0000000000000013 R08: ffffffff82423da0 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> R13: ffff8887f9646480 R14: ffffffff822a7620 R15: ffff8887fae600c0
> FS: 00007f8ee26cd740(0000) GS:ffff8887fecc0000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000000e8 CR3: 00000007f8691000 CR4: 00000000003406e0
> udevd[518]: starting eudev-3.2.9
> rcu: INFO: rcu_sched self-detected stall on CPU
> rcu: 14-....: (20999 ticks this GP) idle=216/1/0x4000000000000002 softirq=454/454 fqs=5250
> (t=21004 jiffies g=-755 q=1327)
> NMI backtrace for cpu 14
> CPU: 14 PID: 520 Comm: pidof Tainted: G D 5.5.7 #1
> Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi, BIOS P3.50 07/18/2019
> Call Trace:
> <IRQ>
> dump_stack+0x50/0x70
> nmi_cpu_backtrace.cold+0x14/0x53
> ? lapic_can_unplug_cpu.cold+0x44/0x44
> nmi_trigger_cpumask_backtrace+0x7b/0x88
> rcu_dump_cpu_stacks+0x7b/0xa9
> rcu_sched_clock_irq.cold+0x152/0x39b
> update_process_times+0x1f/0x50
> tick_sched_timer+0x40/0x90
> ? tick_sched_do_timer+0x50/0x50
> __hrtimer_run_queues+0xdd/0x180
> hrtimer_interrupt+0x108/0x230
> smp_apic_timer_interrupt+0x53/0xa0
> apic_timer_interrupt+0xf/0x20
> </IRQ>
> RIP: 0010:queued_spin_lock_slowpath+0x41/0x1a0
> Code: 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 18 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 84 c0 <75> f8 66 c7 07 01 00 c3 f6 c4 01 75 04 c6 47 01 00 48 c7 c0 40 29
> RSP: 0018:ffffc90001e87d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
> RAX: 0000000000000101 RBX: ffff8887f96f0000 RCX: ffff8887f96f0000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8887f96f0658
> RBP: 000000000000000d R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8887f50b7080
> R13: ffffffff82424480 R14: ffffffff82424480 R15: ffff8887f50b70c0
> ptrace_may_access+0x1e/0x50
> do_task_stat+0x7b/0xd90
> ? do_filp_open+0xab/0x100
> proc_single_show+0x54/0xc0
> ? __kmalloc+0x183/0x210
> seq_read+0xbb/0x3c0
> vfs_read+0xc6/0x150
> ksys_read+0x6b/0x100
> do_syscall_64+0x3d/0x120
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7f666796082e
> Code: c0 e9 f6 fe ff ff 50 48 8d 3d b6 5d 0a 00 e8 e9 fd 01 00 66 0f 1f 84 00 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 0f 05 <48> 3d 00 f0 ff ff 77 5a c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 28
> RSP: 002b:00007ffe26688b38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> RAX: ffffffffffffffda RBX: 00007f6667a58958 RCX: 00007f666796082e
> RDX: 0000000000000400 RSI: 000000000153e590 RDI: 0000000000000004
> RBP: 00007f6667a58950 R08: 00007f6667a36120 R09: 00007ffe266889e0
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
> R13: 0000000000000000 R14: 0000000000000000 R15: 00007ffe2668aee9
> ...
> sysrq: Terminate All Tasks
> rcu: INFO: rcu_sched self-detected stall on CPU
> rcu: 14-....: (83876 ticks this GP) idle=216/1/0x4000000000000002 softirq=454/454 fqs=20970
> (t=84003 jiffies g=-755 q=2695)
> NMI backtrace for cpu 14
> CPU: 14 PID: 520 Comm: pidof Tainted: G D 5.5.7 #1
> Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi, BIOS P3.50 07/18/2019
> Call Trace:
> <IRQ>
> dump_stack+0x50/0x70
> nmi_cpu_backtrace.cold+0x14/0x53
> ? lapic_can_unplug_cpu.cold+0x44/0x44
> nmi_trigger_cpumask_backtrace+0x7b/0x88
> rcu_dump_cpu_stacks+0x7b/0xa9
> rcu_sched_clock_irq.cold+0x152/0x39b
> update_process_times+0x1f/0x50
> tick_sched_timer+0x40/0x90
> ? tick_sched_do_timer+0x50/0x50
> __hrtimer_run_queues+0xdd/0x180
> hrtimer_interrupt+0x108/0x230
> smp_apic_timer_interrupt+0x53/0xa0
> apic_timer_interrupt+0xf/0x20
> </IRQ>
> RIP: 0010:queued_spin_lock_slowpath+0x3d/0x1a0
> Code: 3e f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 18 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 66 c7 07 01 00 c3 f6 c4 01 75 04 c6 47 01 00 48
> RSP: 0018:ffffc90001e87d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
> RAX: 0000000000000101 RBX: ffff8887f96f0000 RCX: ffff8887f96f0000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8887f96f0658
> RBP: 000000000000000d R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8887f50b7080
> R13: ffffffff82424480 R14: ffffffff82424480 R15: ffff8887f50b70c0
> ptrace_may_access+0x1e/0x50
> do_task_stat+0x7b/0xd90
> ? do_filp_open+0xab/0x100
> proc_single_show+0x54/0xc0
> ? __kmalloc+0x183/0x210
> seq_read+0xbb/0x3c0
> vfs_read+0xc6/0x150
> ksys_read+0x6b/0x100
> do_syscall_64+0x3d/0x120
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7f666796082e
> Code: c0 e9 f6 fe ff ff 50 48 8d 3d b6 5d 0a 00 e8 e9 fd 01 00 66 0f 1f 84 00 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 0f 05 <48> 3d 00 f0 ff ff 77 5a c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 28
> RSP: 002b:00007ffe26688b38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> RAX: ffffffffffffffda RBX: 00007f6667a58958 RCX: 00007f666796082e
> RDX: 0000000000000400 RSI: 000000000153e590 RDI: 0000000000000004
> RBP: 00007f6667a58950 R08: 00007f6667a36120 R09: 00007ffe266889e0
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
> R13: 0000000000000000 R14: 0000000000000000 R15: 00007ffe2668aee9
> rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 14-... } 21041 jiffies s: 45 root: 0x4000/.
> rcu: blocking rcu_node structures:
> Task dump for CPU 14:
> pidof R running task 0 520 1 0x8000000c
> Call Trace:
> ? do_syscall_64+0x3d/0x120
> ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
> rcu: INFO: rcu_sched self-detected stall on CPU
> rcu: 14-....: (146878 ticks this GP) idle=216/1/0x4000000000000002 softirq=454/454 fqs=36715
> (t=147006 jiffies g=-755 q=3376)
> NMI backtrace for cpu 14
> CPU: 14 PID: 520 Comm: pidof Tainted: G D 5.5.7 #1
> Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi, BIOS P3.50 07/18/2019
> Call Trace:
> <IRQ>
> dump_stack+0x50/0x70
> nmi_cpu_backtrace.cold+0x14/0x53
> ? lapic_can_unplug_cpu.cold+0x44/0x44
> nmi_trigger_cpumask_backtrace+0x7b/0x88
> rcu_dump_cpu_stacks+0x7b/0xa9
> rcu_sched_clock_irq.cold+0x152/0x39b
> update_process_times+0x1f/0x50
> tick_sched_timer+0x40/0x90
> ? tick_sched_do_timer+0x50/0x50
> __hrtimer_run_queues+0xdd/0x180
> hrtimer_interrupt+0x108/0x230
> smp_apic_timer_interrupt+0x53/0xa0
> apic_timer_interrupt+0xf/0x20
> </IRQ>
> RIP: 0010:queued_spin_lock_slowpath+0x3d/0x1a0
> Code: 3e f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 18 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 66 c7 07 01 00 c3 f6 c4 01 75 04 c6 47 01 00 48
> RSP: 0018:ffffc90001e87d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
> RAX: 0000000000000101 RBX: ffff8887f96f0000 RCX: ffff8887f96f0000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8887f96f0658
> RBP: 000000000000000d R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8887f50b7080
> R13: ffffffff82424480 R14: ffffffff82424480 R15: ffff8887f50b70c0
> ptrace_may_access+0x1e/0x50
> do_task_stat+0x7b/0xd90
> ? do_filp_open+0xab/0x100
> proc_single_show+0x54/0xc0
> ? __kmalloc+0x183/0x210
> seq_read+0xbb/0x3c0
> vfs_read+0xc6/0x150
> ksys_read+0x6b/0x100
> do_syscall_64+0x3d/0x120
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7f666796082e
> Code: c0 e9 f6 fe ff ff 50 48 8d 3d b6 5d 0a 00 e8 e9 fd 01 00 66 0f 1f 84 00 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 0f 05 <48> 3d 00 f0 ff ff 77 5a c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 28
> RSP: 002b:00007ffe26688b38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> RAX: ffffffffffffffda RBX: 00007f6667a58958 RCX: 00007f666796082e
> RDX: 0000000000000400 RSI: 000000000153e590 RDI: 0000000000000004
> RBP: 00007f6667a58950 R08: 00007f6667a36120 R09: 00007ffe266889e0
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
> R13: 0000000000000000 R14: 0000000000000000 R15: 00007ffe2668aee9
> rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 14-... } 85041 jiffies s: 45 root: 0x4000/.
> rcu: blocking rcu_node structures:
> Task dump for CPU 14:
> pidof R running task 0 520 1 0x8000000c
> Call Trace:
> ? do_syscall_64+0x3d/0x120
> ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
> rcu: INFO: rcu_sched self-detected stall on CPU
> rcu: 14-....: (209792 ticks this GP) idle=216/1/0x4000000000000002 softirq=454/454 fqs=52439
> (t=210009 jiffies g=-755 q=3747)
> NMI backtrace for cpu 14
> CPU: 14 PID: 520 Comm: pidof Tainted: G D 5.5.7 #1
> Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi, BIOS P3.50 07/18/2019
> Call Trace:
> <IRQ>
> dump_stack+0x50/0x70
> nmi_cpu_backtrace.cold+0x14/0x53
> ? lapic_can_unplug_cpu.cold+0x44/0x44
> nmi_trigger_cpumask_backtrace+0x7b/0x88
> rcu_dump_cpu_stacks+0x7b/0xa9
> rcu_sched_clock_irq.cold+0x152/0x39b
> update_process_times+0x1f/0x50
> tick_sched_timer+0x40/0x90
> ? tick_sched_do_timer+0x50/0x50
> __hrtimer_run_queues+0xdd/0x180
> hrtimer_interrupt+0x108/0x230
> smp_apic_timer_interrupt+0x53/0xa0
> apic_timer_interrupt+0xf/0x20
> </IRQ>
> RIP: 0010:queued_spin_lock_slowpath+0x3d/0x1a0
> Code: 3e f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 18 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 66 c7 07 01 00 c3 f6 c4 01 75 04 c6 47 01 00 48
> RSP: 0018:ffffc90001e87d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
> RAX: 0000000000000101 RBX: ffff8887f96f0000 RCX: ffff8887f96f0000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8887f96f0658
> RBP: 000000000000000d R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8887f50b7080
> R13: ffffffff82424480 R14: ffffffff82424480 R15: ffff8887f50b70c0
> ptrace_may_access+0x1e/0x50
> do_task_stat+0x7b/0xd90
> ? do_filp_open+0xab/0x100
> proc_single_show+0x54/0xc0
> ? __kmalloc+0x183/0x210
> seq_read+0xbb/0x3c0
> vfs_read+0xc6/0x150
> ksys_read+0x6b/0x100
> do_syscall_64+0x3d/0x120
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7f666796082e
> Code: c0 e9 f6 fe ff ff 50 48 8d 3d b6 5d 0a 00 e8 e9 fd 01 00 66 0f 1f 84 00 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 0f 05 <48> 3d 00 f0 ff ff 77 5a c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 28
> RSP: 002b:00007ffe26688b38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> RAX: ffffffffffffffda RBX: 00007f6667a58958 RCX: 00007f666796082e
> RDX: 0000000000000400 RSI: 000000000153e590 RDI: 0000000000000004
> RBP: 00007f6667a58950 R08: 00007f6667a36120 R09: 00007ffe266889e0
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
> R13: 0000000000000000 R14: 0000000000000000 R15: 00007ffe2668aee9
> rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 14-... } 150577 jiffies s: 45 root: 0x4000/.
> rcu: blocking rcu_node structures:
> Task dump for CPU 14:
> pidof R running task 0 520 1 0x8000000c
> Call Trace:
> ? do_syscall_64+0x3d/0x120
> ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
>
>

\
 
 \ /
  Last update: 2020-03-05 07:11    [W:0.073 / U:31.744 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site