Messages in this thread | | | From | Hao Sun <> | Date | Mon, 20 Sep 2021 21:08:08 +0800 | Subject | possible deadlock in __btrfs_release_delayed_node |
| |
Hello,
When using Healer to fuzz the latest Linux kernel, the following crash was triggered.
HEAD commit: 4357f03d6611 Merge tag 'pm-5.15-rc2 git tree: upstream console output: https://drive.google.com/file/d/1Is2XjAwyco_VmymNbbsMH4xZysHsgfag/view?usp=sharing kernel config: https://drive.google.com/file/d/1HKZtF_s3l6PL3OoQbNq_ei9CdBus-Tz0/view?usp=sharing
Sorry, I don't have a reproducer for this crash, hope the symbolized report can help. If you fix this issue, please add the following tag to the commit: Reported-by: Hao Sun <sunhao.th@gmail.com>
loop9: detected capacity change from 0 to 32768 BTRFS info (device loop9): disk space caching is enabled BTRFS info (device loop9): has skinny extents BTRFS info (device loop9): enabling ssd optimizations FAULT_INJECTION: forcing a failure. name failslab, interval 1, probability 0, space 0, times 0 CPU: 3 PID: 16884 Comm: syz-executor Not tainted 5.15.0-rc1+ #19 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x8d/0xcf lib/dump_stack.c:106 fail_dump lib/fault-inject.c:52 [inline] should_fail+0x13c/0x160 lib/fault-inject.c:146 should_failslab+0x5/0x10 mm/slab_common.c:1328 slab_pre_alloc_hook.constprop.99+0x4e/0xc0 mm/slab.h:494 slab_alloc_node mm/slub.c:3120 [inline] slab_alloc mm/slub.c:3214 [inline] kmem_cache_alloc+0x44/0x280 mm/slub.c:3219 btrfs_add_delayed_tree_ref+0xc3/0x580 fs/btrfs/delayed-ref.c:917 btrfs_alloc_tree_block+0x478/0x670 fs/btrfs/extent-tree.c:4853 __btrfs_cow_block+0x16f/0x820 fs/btrfs/ctree.c:415 btrfs_cow_block+0x12a/0x300 fs/btrfs/ctree.c:570 btrfs_search_slot+0x6b0/0xee0 fs/btrfs/ctree.c:1768 btrfs_lookup_inode+0x50/0x110 fs/btrfs/inode-item.c:408 __btrfs_update_delayed_inode+0x81/0x3a0 fs/btrfs/delayed-inode.c:948 btrfs_commit_inode_delayed_inode+0x14a/0x160 fs/btrfs/delayed-inode.c:1189 btrfs_log_inode+0x438/0x1ab0 fs/btrfs/tree-log.c:5385 btrfs_log_inode_parent+0x272/0x1110 fs/btrfs/tree-log.c:6168 btrfs_log_dentry_safe+0x3a/0x50 fs/btrfs/tree-log.c:6269 btrfs_sync_file+0x2cd/0x760 fs/btrfs/file.c:2264 vfs_fsync_range+0x48/0xa0 fs/sync.c:200 generic_write_sync include/linux/fs.h:2955 [inline] btrfs_file_write_iter+0x34f/0x510 fs/btrfs/file.c:2034 call_write_iter include/linux/fs.h:2163 [inline] new_sync_write+0x18d/0x260 fs/read_write.c:507 vfs_write+0x43b/0x4a0 fs/read_write.c:594 ksys_write+0xd2/0x120 fs/read_write.c:647 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x34/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x46ae99 Code: f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f685738cc48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 000000000078c0a0 RCX: 000000000046ae99 RDX: 000000000000000b RSI: 0000000020005940 RDI: 0000000000000003 RBP: 00007f685738cc80 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000001e R13: 0000000000000000 R14: 000000000078c0a0 R15: 00007ffc71ddb640 BTRFS: error (device loop9) in __btrfs_update_delayed_inode:995: errno=-12 Out of memory BTRFS info (device loop9): forced readonly
====================================================== WARNING: possible circular locking dependency detected 5.15.0-rc1+ #19 Not tainted ------------------------------------------------------ syz-executor/16884 is trying to acquire lock: ffff88810f946ab0 (&delayed_node->mutex){+.+.}-{3:3}, at: __btrfs_release_delayed_node+0x4e/0x410 fs/btrfs/delayed-inode.c:262
but task is already holding lock: ffff888016e696e8 (btrfs-tree-01/1){+.+.}-{3:3}, at: __btrfs_tree_lock+0x2e/0x1a0 fs/btrfs/locking.c:112
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (btrfs-tree-01/1){+.+.}-{3:3}: __lock_release kernel/locking/lockdep.c:5341 [inline] lock_release+0x135/0x2e0 kernel/locking/lockdep.c:5645 up_write+0x12/0x130 kernel/locking/rwsem.c:1569 __btrfs_cow_block+0x552/0x820 fs/btrfs/ctree.c:491 btrfs_cow_block+0x12a/0x300 fs/btrfs/ctree.c:570 btrfs_search_slot+0x6b0/0xee0 fs/btrfs/ctree.c:1768 btrfs_insert_empty_items+0x80/0xf0 fs/btrfs/ctree.c:3905 btrfs_insert_delayed_item fs/btrfs/delayed-inode.c:736 [inline] btrfs_insert_delayed_items fs/btrfs/delayed-inode.c:783 [inline] __btrfs_commit_inode_delayed_items fs/btrfs/delayed-inode.c:1041 [inline] __btrfs_run_delayed_items+0x307/0x6a0 fs/btrfs/delayed-inode.c:1083 btrfs_commit_transaction+0x268/0x1450 fs/btrfs/transaction.c:2170 btrfs_sync_fs+0x9a/0x430 fs/btrfs/super.c:1426 __sync_filesystem fs/sync.c:39 [inline] sync_filesystem+0x9b/0xd0 fs/sync.c:67 generic_shutdown_super+0x30/0x170 fs/super.c:448 kill_anon_super+0x15/0x30 fs/super.c:1057 btrfs_kill_super+0x19/0x30 fs/btrfs/super.c:2348 deactivate_locked_super+0x43/0x80 fs/super.c:335 deactivate_super+0x53/0x80 fs/super.c:366 cleanup_mnt+0x138/0x1b0 fs/namespace.c:1137 task_work_run+0x86/0xd0 kernel/task_work.c:164 tracehook_notify_resume include/linux/tracehook.h:189 [inline] exit_to_user_mode_loop kernel/entry/common.c:175 [inline] exit_to_user_mode_prepare+0x271/0x280 kernel/entry/common.c:209 __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline] syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:302 do_syscall_64+0x40/0xb0 arch/x86/entry/common.c:86 entry_SYSCALL_64_after_hwframe+0x44/0xae
-> #0 (&delayed_node->mutex){+.+.}-{3:3}: check_prev_add kernel/locking/lockdep.c:3051 [inline] check_prevs_add kernel/locking/lockdep.c:3174 [inline] validate_chain kernel/locking/lockdep.c:3789 [inline] __lock_acquire+0x161f/0x1d60 kernel/locking/lockdep.c:5015 lock_acquire+0x1f9/0x340 kernel/locking/lockdep.c:5625 __mutex_lock_common kernel/locking/mutex.c:596 [inline] __mutex_lock+0x98/0xa50 kernel/locking/mutex.c:729 __btrfs_release_delayed_node+0x4e/0x410 fs/btrfs/delayed-inode.c:262 btrfs_release_delayed_node fs/btrfs/delayed-inode.c:287 [inline] btrfs_commit_inode_delayed_inode+0x104/0x160 fs/btrfs/delayed-inode.c:1201 btrfs_log_inode+0x438/0x1ab0 fs/btrfs/tree-log.c:5385 btrfs_log_inode_parent+0x272/0x1110 fs/btrfs/tree-log.c:6168 btrfs_log_dentry_safe+0x3a/0x50 fs/btrfs/tree-log.c:6269 btrfs_sync_file+0x2cd/0x760 fs/btrfs/file.c:2264 vfs_fsync_range+0x48/0xa0 fs/sync.c:200 generic_write_sync include/linux/fs.h:2955 [inline] btrfs_file_write_iter+0x34f/0x510 fs/btrfs/file.c:2034 call_write_iter include/linux/fs.h:2163 [inline] new_sync_write+0x18d/0x260 fs/read_write.c:507 vfs_write+0x43b/0x4a0 fs/read_write.c:594 ksys_write+0xd2/0x120 fs/read_write.c:647 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x34/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1 ---- ---- lock(btrfs-tree-01/1); lock(&delayed_node->mutex); lock(btrfs-tree-01/1); lock(&delayed_node->mutex);
*** DEADLOCK ***
6 locks held by syz-executor/16884: #0: ffff888104f336f0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x55/0x60 fs/file.c:990 #1: ffff88811394b460 (sb_writers#18){.+.+}-{0:0}, at: ksys_write+0xd2/0x120 fs/read_write.c:647 #2: ffff888110cfab70 (&sb->s_type->i_mutex_key#26){++++}-{3:3}, at: inode_lock include/linux/fs.h:786 [inline] #2: ffff888110cfab70 (&sb->s_type->i_mutex_key#26){++++}-{3:3}, at: btrfs_inode_lock+0x65/0xb0 fs/btrfs/inode.c:126 #3: ffff888110cfa9f8 (&ei->i_mmap_lock){+.+.}-{3:3}, at: btrfs_inode_lock+0x80/0xb0 fs/btrfs/inode.c:129 #4: ffff88811394b650 (sb_internal#2){.+.+}-{0:0}, at: btrfs_sync_file+0x29e/0x760 fs/btrfs/file.c:2257 #5: ffff888016e696e8 (btrfs-tree-01/1){+.+.}-{3:3}, at: __btrfs_tree_lock+0x2e/0x1a0 fs/btrfs/locking.c:112
stack backtrace: CPU: 1 PID: 16884 Comm: syz-executor Not tainted 5.15.0-rc1+ #19 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x8d/0xcf lib/dump_stack.c:106 check_noncircular+0x105/0x120 kernel/locking/lockdep.c:2131 check_prev_add kernel/locking/lockdep.c:3051 [inline] check_prevs_add kernel/locking/lockdep.c:3174 [inline] validate_chain kernel/locking/lockdep.c:3789 [inline] __lock_acquire+0x161f/0x1d60 kernel/locking/lockdep.c:5015 lock_acquire+0x1f9/0x340 kernel/locking/lockdep.c:5625 __mutex_lock_common kernel/locking/mutex.c:596 [inline] __mutex_lock+0x98/0xa50 kernel/locking/mutex.c:729 __btrfs_release_delayed_node+0x4e/0x410 fs/btrfs/delayed-inode.c:262 btrfs_release_delayed_node fs/btrfs/delayed-inode.c:287 [inline] btrfs_commit_inode_delayed_inode+0x104/0x160 fs/btrfs/delayed-inode.c:1201 btrfs_log_inode+0x438/0x1ab0 fs/btrfs/tree-log.c:5385 btrfs_log_inode_parent+0x272/0x1110 fs/btrfs/tree-log.c:6168 btrfs_log_dentry_safe+0x3a/0x50 fs/btrfs/tree-log.c:6269 btrfs_sync_file+0x2cd/0x760 fs/btrfs/file.c:2264 vfs_fsync_range+0x48/0xa0 fs/sync.c:200 generic_write_sync include/linux/fs.h:2955 [inline] btrfs_file_write_iter+0x34f/0x510 fs/btrfs/file.c:2034 call_write_iter include/linux/fs.h:2163 [inline] new_sync_write+0x18d/0x260 fs/read_write.c:507 vfs_write+0x43b/0x4a0 fs/read_write.c:594 ksys_write+0xd2/0x120 fs/read_write.c:647 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x34/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x46ae99 Code: f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f685738cc48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 000000000078c0a0 RCX: 000000000046ae99 RDX: 000000000000000b RSI: 0000000020005940 RDI: 0000000000000003 RBP: 00007f685738cc80 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000001e R13: 0000000000000000 R14: 000000000078c0a0 R15: 00007ffc71ddb640
| |