Messages in this thread | | | From | Dmitry Vyukov <> | Date | Tue, 24 Jan 2017 14:37:28 +0100 | Subject | Re: x86: WARNING: kernel stack frame pointer in kworker has bad value |
| |
On Mon, Jan 23, 2017 at 11:50 PM, Josh Poimboeuf <jpoimboe@redhat.com> wrote: > On Mon, Jan 23, 2017 at 07:43:18PM +0100, Dmitry Vyukov wrote: >> Hello, >> >> I am getting the following warnings while running syzkaller fuzzer on >> 7a308bb3016f57e5be11a677d15b821536419d36: >> >> WARNING: kernel stack frame pointer at ffff880065f7fef8 in >> kworker/1:3:21075 has bad value ffffffff81598080 >> unwind stack type:0 next_sp: (null) mask:2 graph_idx:0 >> ... >> >> Here are 4 examples: >> >> https://gist.githubusercontent.com/dvyukov/7258fca11e5c3db6e8f8fb684e6168ff/raw/2ca14f3c7294c7e466611f8ecfb3072c676a657c/gistfile1.txt >> https://gist.githubusercontent.com/dvyukov/cc442b144ae712f0e1bdeb065085e36a/raw/f713cef7e9689914fc94838a0a7d4192224fb6d2/gistfile1.txt >> https://gist.githubusercontent.com/dvyukov/9058cce2647e3d37157c283c6400498c/raw/7a8ec8d71b9e95eeb24156073eeef96ba61d95ea/gistfile1.txt >> https://gist.githubusercontent.com/dvyukov/b899dfa72d611b511a674b63676921f3/raw/90203ed26b6674dedba8c1c16dd76309ef21fbea/gistfile1.txt >> >> Unfortunately this is not reproducible, but there is a stack dump, so >> maybe it's enough to figure out the root cause. > > This seems like some kind of stack corruption. Here are the stack > traces: > >> https://gist.githubusercontent.com/dvyukov/7258fca11e5c3db6e8f8fb684e6168ff/raw/2ca14f3c7294c7e466611f8ecfb3072c676a657c/gistfile1.txt > > ffff8800520eef68: ffffffff812b148e (__save_stack_trace+0x6e/0xd0) > ffff8800520eefe8: ffffffff812b1506 (save_stack_trace+0x16/0x20) > ffff8800520eeff8: ffffffff81a0c843 (save_stack+0x43/0xd0) > ffff8800520ef228: ffffffff81a0d17f (kasan_slab_free+0x6f/0xb0) > ffff8800520ef250: ffffffff81a0a4e1 (kmem_cache_free+0x71/0x240) > ffff8800520ef280: ffffffff822924d4 (ioc_release_fn+0x2e4/0x380) > ffff8800520ef550: ffffffff814b4900 (process_one_work+0xbd0/0x1c10) > ffff8800520ef9a8: ffffffff814b5b63 (worker_thread+0x223/0x1990) > ffff8800520eff00: ffffffff814ce526 (kthread+0x326/0x3f0) > >> https://gist.githubusercontent.com/dvyukov/cc442b144ae712f0e1bdeb065085e36a/raw/f713cef7e9689914fc94838a0a7d4192224fb6d2/gistfile1.txt > > ffff88005b40ef68: ffffffff812b260e (__save_stack_trace+0x6e/0xd0) > ffff88005b40efe8: ffffffff812b2686 (save_stack_trace+0x16/0x20) > ffff88005b40eff8: ffffffff81a0e8c3 (save_stack+0x43/0xd0) > ffff88005b40f228: ffffffff81a0f1ff (kasan_slab_free+0x6f/0xb0) > ffff88005b40f250: ffffffff81a0c561 (kmem_cache_free+0x71/0x240) > ffff88005b40f280: ffffffff822943b4 (ioc_release_fn+0x2e4/0x380) > ffff88005b40f550: ffffffff814b6980 (process_one_work+0xbd0/0x1c10) > ffff88005b40f9a8: ffffffff814b7be3 (worker_thread+0x223/0x1990) > ffff88005b40ff00: ffffffff814d05a6 (kthread+0x326/0x3f0) > >> https://gist.githubusercontent.com/dvyukov/9058cce2647e3d37157c283c6400498c/raw/7a8ec8d71b9e95eeb24156073eeef96ba61d95ea/gistfile1.txt > > ffff880065f7ef68: ffffffff812b260e (__save_stack_trace+0x6e/0xd0) > ffff880065f7efe8: ffffffff812b2686 (save_stack_trace+0x16/0x20) > ffff880065f7eff8: ffffffff81a0e8c3 (save_stack+0x43/0xd0) > ffff880065f7f228: ffffffff81a0f1ff (kasan_slab_free+0x6f/0xb0) > ffff880065f7f250: ffffffff81a0c561 (kmem_cache_free+0x71/0x240) > ffff880065f7f280: ffffffff822943b4 (ioc_release_fn+0x2e4/0x380) > ffff880065f7f550: ffffffff814b6980 (process_one_work+0xbd0/0x1c10) > ffff880065f7f9a8: ffffffff814b7be3 (worker_thread+0x223/0x1990) > ffff880065f7ff00: ffffffff814d05a6 (kthread+0x326/0x3f0) > >> https://gist.githubusercontent.com/dvyukov/b899dfa72d611b511a674b63676921f3/raw/90203ed26b6674dedba8c1c16dd76309ef21fbea/gistfile1.txt > > ffff880065e57220: ffffffff812b148e (__save_stack_trace+0x6e/0xd0) > ffff880065e572a0: ffffffff812b1506 (save_stack_trace+0x16/0x20) > ffff880065e572b0: ffffffff81a0c843 (save_stack+0x43/0xd0) > ffff880065e574e0: ffffffff81a0cb0a (kasan_kmalloc+0xaa/0xd0) > ffff880065e57520: ffffffff81a0d102 (kasan_slab_alloc+0x12/0x20) > ffff880065e57530: ffffffff81a08704 (kmem_cache_alloc+0xf4/0x680) > ffff880065e57598: ffffffff823293b3 (ida_pre_get+0x1f3/0x3b0) > ffff880065e57720: ffffffff823296d8 (ida_simple_get+0x168/0x320) > ffff880065e578e8: ffffffff814ad926 (create_worker+0x96/0x620) > ffff880065e579a8: ffffffff814b68ad (worker_thread+0xf6d/0x1990) > ffff880065e57f00: ffffffff814ce526 (kthread+0x326/0x3f0) > > In all four cases, the frame pointer pushed by worker_thread() is > somehow changed to be the address at the end of > debug_check_no_locks_freed(): > > ffff880065f7fef8: ffffffff81598080 (debug_check_no_locks_freed+0x5c0/0x5c0) > > Can you please share the following: > > - The output of "grep ffffffff81598080 System.map" > - The .config file
Hi,
Thanks for looking at this.
I've attached my config. Unfortunately I don't have that System.map/vmlinux, I've rebased my kernel and rebuilt it. But here is what after debug_check_no_locks_freed how:
vmlinux:ffffffff815956a0 0000000000000467 T lockdep_count_backward_deps vmlinux:ffffffff81595b10 00000000000005b6 T debug_check_no_locks_freed vmlinux:ffffffff815960d0 0000000000003428 t __lock_acquire
But If you suspect a corruption, then I would suggest to stop spending more time on this now. At least until I have more data. This happened while I stress tested KVM, and I've seen a low rate of what looks like random memory corruptions (e.g. corrupted slab freelists and other unexplainable things). So let's write it off onto KVM for now. [unhandled content-type:application/octet-stream] | |