Messages in this thread | | | Date | Thu, 11 Aug 2022 20:33:58 -0700 | Subject | Re: Kernel Panic in skb_release_data using genet | From | Florian Fainelli <> |
| |
On 5/17/2022 12:52 AM, Maxime Ripard wrote: > It's not really 100% reliable, but happens 30%-50% of the time at boot > when KASAN is enabled. It seems like enabling KASAN increases that > likelihood though, it went unnoticed for some time before I started > having those issues again when I enabled it for something unrelated. > > It looks like it happens in bursts though, so I would get 10-15 boots > fine, and then 4-5 boots with that crash. > > Cold boot vs reboot doesn't seem to affect it in one way or the other. > >> What version of GCC did you build your kernel with? > > The arm64 cross-compiler packaged by Fedora, which is GCC 11.2 > at the moment. > >> How often does that happen? What config.txt file are you using >> for your Pi4 B? > > You'll find my config.txt and kernel .config attached
OK, so this is what I have been able to reproduce so far but this does not appear to be very reliable to reproduce, I will try my best to hold on to that lead though, thanks for your patience.
# udhcpc -i eth0 udhcpc: started, v1.35.0 [ 34.355086] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay) [ 34.363758] ================================================================== [ 34.371106] BUG: KASAN: user-memory-access in put_page+0x10/0x64 [ 34.377227] Read of size 4 at addr 01000085 by task ifconfig/165 [ 34.383338] [ 34.384857] CPU: 0 PID: 165 Comm: ifconfig Tainted: G W 5.19.0 #43 [ 34.392560] Hardware name: BCM2711 [ 34.396020] unwind_backtrace from show_stack+0x18/0x1c [ 34.401354] show_stack from dump_stack_lvl+0x40/0x4c [ 34.406502] dump_stack_lvl from kasan_report+0x8c/0xa4 [ 34.411825] kasan_report from put_page+0x10/0x64 [ 34.416615] put_page from skb_release_data+0x84/0x13c [ 34.421847] skb_release_data from __kfree_skb+0x14/0x20 [ 34.427256] __kfree_skb from bcmgenet_rx_poll+0x504/0x6f8 [ 34.432846] bcmgenet_rx_poll from __napi_poll.constprop.0+0x50/0x1c0 [ 34.439407] __napi_poll.constprop.0 from net_rx_action+0x278/0x488 [ 34.445787] net_rx_action from __do_softirq+0x268/0x390 [ 34.451197] __do_softirq from __irq_exit_rcu+0x88/0xf8 [ 34.456521] __irq_exit_rcu from irq_exit+0x10/0x18 [ 34.461492] irq_exit from call_with_stack+0x18/0x20 [ 34.466553] call_with_stack from __irq_svc+0x84/0x94 [ 34.471696] Exception stack(0xf0d337f8 to 0xf0d33840) [ 34.476835] 37e0: c5548580 00000003 [ 34.485156] 3800: 00002000 f0a40808 c5548000 c5548580 00000000 c554b000 c5548580 c554bdd0 [ 34.493474] 3820: 00000000 00000004 c5548580 f0d33848 c094329c c09432bc 00070013 ffffffff [ 34.501788] __irq_svc from bcmgenet_open+0xe1c/0x1094 [ 34.507023] bcmgenet_open from __dev_open+0x1e4/0x21c [ 34.512258] __dev_open from __dev_change_flags+0x228/0x25c [ 34.517931] __dev_change_flags from dev_change_flags+0x48/0x88 [ 34.523958] dev_change_flags from devinet_ioctl+0x3ac/0x834 [ 34.529723] devinet_ioctl from inet_ioctl+0x250/0x2a4 [ 34.534956] inet_ioctl from sock_ioctl+0x1dc/0x410 [ 34.539927] sock_ioctl from vfs_ioctl+0x50/0x64 [ 34.544632] vfs_ioctl from sys_ioctl+0x134/0xa7c [ 34.549422] sys_ioctl from ret_fast_syscall+0x0/0x4c [ 34.554565] Exception stack(0xf0d33fa8 to 0xf0d33ff0) [ 34.559705] 3fa0: 0051fd98 0053f9dc 00000003 00008914 b6dc5c4c b6dc5bd0 [ 34.568025] 3fc0: 0051fd98 0053f9dc b6dc5f55 00000036 b6dc5e48 00000003 aed11d00 aed12010 [ 34.576341] 3fe0: 00000036 b6dc5bb8 aec4c2f3 aebdda66 [ 34.581475] ================================================================== [ 34.588882] Disabling lock debugging due to kernel taint [ 34.594288] 8<--- cut here --- [ 34.597412] Unable to handle kernel paging request at virtual address 01000085 [ 34.604775] [01000085] *pgd=01982003, *pmd=00000000 [ 34.609751] Internal error: Oops: 206 [#1] SMP ARM [ 34.614624] Modules linked in: [ 34.617734] CPU: 0 PID: 165 Comm: ifconfig Tainted: G B W 5.19.0 #43 [ 34.625435] Hardware name: BCM2711 [ 34.628892] PC is at put_page+0x14/0x64 [ 34.632800] LR is at kasan_report+0x98/0xa4 [ 34.637056] pc : [<c0b4bee4>] lr : [<c047ea5c>] psr: 60070113 [ 34.643427] sp : f0803d50 ip : 00000000 fp : c554bfd8 [ 34.648739] r10: 00007f5e r9 : c694f582 r8 : c1fef15e [ 34.654052] r7 : c694f5b8 r6 : c694f580 r5 : 01000081 r4 : c1fef100 [ 34.660689] r3 : 00000000 r2 : c1f047c0 r1 : 00000004 r0 : 00000001 [ 34.667325] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [ 34.674582] Control: 30c5383d Table: 0606b700 DAC: fffffffd [ 34.680422] Register r0 information: non-paged memory [ 34.685565] Register r1 information: non-paged memory [ 34.690705] Register r2 information: slab task_struct start c1f047c0 pointer offset 0 [ 34.698690] Register r3 information: NULL pointer [ 34.703477] Register r4 information: slab skbuff_head_cache start c1fef100 pointer offset 0 size 48 [ 34.712699] Register r5 information: non-paged memory [ 34.717839] Register r6 information: non-slab/vmalloc memory [ 34.723595] Register r7 information: non-slab/vmalloc memory [ 34.729352] Register r8 information: slab skbuff_head_cache start c1fef100 pointer offset 94 size 48 [ 34.738662] Register r9 information: non-slab/vmalloc memory [ 34.744419] Register r10 information: non-paged memory [ 34.749646] Register r11 information: non-slab/vmalloc memory [ 34.755492] Register r12 information: NULL pointer [ 34.760366] Process ifconfig (pid: 165, stack limit = 0xf517d551) [ 34.766573] Stack: (0xf0803d50 to 0xf0804000) [ 34.771005] 3d40: c1fef100 00000001 c694f580 c0b4dc74 [ 34.779325] 3d60: c1fef100 c5548000 c5548580 c1fef100 f0803e40 7f5e0001 00007f5e c0b4db24 [ 34.787644] 3d80: c554bdd0 c0940f84 0bc80000 b4c23195 c2cb12c0 c0efdab0 c2cb12c0 00000001 [ 34.795963] 3da0: 00000000 00000040 00000004 c554bec4 1e1007bc c554beb8 c5548588 00000004 [ 34.804282] 3dc0: c55498bc c554bec8 c02d5684 00000003 00000000 c02b6e10 e7df0980 c02bf390 [ 34.812601] 3de0: 41b58ab3 c15fec7a c0940a80 c1f047c0 00070113 257ac000 e7de97cc ffff982d [ 34.820919] 3e00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 b4c23195 [ 34.829237] 3e20: c1f047c0 e7de8680 00000000 c1f047c0 00000000 c076733c e7de9ad8 00000000 [ 34.837556] 3e40: e7de97d4 c613e0a0 00000001 c554bdd0 00000001 00000040 f0803ef0 c554bdd8 [ 34.845875] 3e60: 257ac000 c2805d40 e7df0d00 c0b70f24 c554bdd0 f0803ef0 00000000 e7df0b40 [ 34.854195] 3e80: f0803f60 bd1007d8 c554bdd0 c2644b40 257ac000 c0b7130c 0000012c e7df0d0c [ 34.862513] 3ea0: ffff9839 f0803ef0 81d99054 c554bdd4 0000002c 257ac000 c26433c8 c0840554 [ 34.870832] 3ec0: 41b58ab3 c1612850 c0b71094 c2cb12c0 e7df0980 c02d8a5c ea8ed400 c02d8ae0 [ 34.879150] 3ee0: 41b58ab3 c15f3580 c08403c4 00000010 c554bd00 c554bdd8 00000000 00000010 [ 34.887470] 3f00: f0803f00 f0803f00 c5548580 00002000 c554bdd0 c554b580 0000010a c093e0b8 [ 34.895788] 3f20: f0803f20 f0803f20 0000002c c093df98 c2806f18 c029f4ac 00000000 00000007 [ 34.904108] 3f40: e7de9780 c02a4218 00000104 c4dca800 00000001 c4dca824 c4dca86c c4dca86c [ 34.912427] 3f60: c4dca848 f0803fc8 f0d337f0 b4c23195 c4dca800 c1f047c0 c280508c 00000008 [ 34.920747] 3f80: c2643dc0 c1f047c4 00000003 00000100 c1f049d4 c02014d8 c4dca800 c1f047c0 [ 34.929066] 3fa0: 00400100 0000000a ffff9838 00000004 c263c3c8 257ac000 c26433c0 c1f047c0 [ 34.937385] 3fc0: c2643dc0 c1f047c4 257ac000 257ac000 c1f047c0 00000000 f0d337f0 c02312c4 [ 34.945704] 3fe0: c09432bc 00070013 ffffffff f0d3382c c5548580 c0231418 c09432bc c07559fc [ 34.954019] put_page from skb_release_data+0x84/0x13c [ 34.959252] skb_release_data from __kfree_skb+0x14/0x20 [ 34.964660] __kfree_skb from bcmgenet_rx_poll+0x504/0x6f8 [ 34.970250] bcmgenet_rx_poll from __napi_poll.constprop.0+0x50/0x1c0 [ 34.976812] __napi_poll.constprop.0 from net_rx_action+0x278/0x488 [ 34.983192] net_rx_action from __do_softirq+0x268/0x390 [ 34.988602] __do_softirq from __irq_exit_rcu+0x88/0xf8 [ 34.993927] __irq_exit_rcu from irq_exit+0x10/0x18 [ 34.998899] irq_exit from call_with_stack+0x18/0x20 [ 35.003958] call_with_stack from __irq_svc+0x84/0x94 [ 35.009101] Exception stack(0xf0d337f8 to 0xf0d33840) [ 35.014238] 37e0: c5548580 00000003 [ 35.022557] 3800: 00002000 f0a40808 c5548000 c5548580 00000000 c554b000 c5548580 c554bdd0 [ 35.030877] 3820: 00000000 00000004 c5548580 f0d33848 c094329c c09432bc 00070013 ffffffff [ 35.039192] __irq_svc from bcmgenet_open+0xe1c/0x1094 [ 35.044427] bcmgenet_open from __dev_open+0x1e4/0x21c [ 35.049661] __dev_open from __dev_change_flags+0x228/0x25c [ 35.055334] __dev_change_flags from dev_change_flags+0x48/0x88 [ 35.061361] dev_change_flags from devinet_ioctl+0x3ac/0x834 [ 35.067125] devinet_ioctl from inet_ioctl+0x250/0x2a4 [ 35.072359] inet_ioctl from sock_ioctl+0x1dc/0x410 [ 35.077330] sock_ioctl from vfs_ioctl+0x50/0x64 [ 35.082034] vfs_ioctl from sys_ioctl+0x134/0xa7c [ 35.086825] sys_ioctl from ret_fast_syscall+0x0/0x4c [ 35.091969] Exception stack(0xf0d33fa8 to 0xf0d33ff0) [ 35.097109] 3fa0: 0051fd98 0053f9dc 00000003 00008914 b6dc5c4c b6dc5bd0 [ 35.105428] 3fc0: 0051fd98 0053f9dc b6dc5f55 00000036 b6dc5e48 00000003 aed11d00 aed12010 [ 35.113744] 3fe0: 00000036 b6dc5bb8 aec4c2f3 aebdda66 [ 35.118883] Code: e1a05000 e2800004 ebe4cca7 e3a01004 (e5953004) [ 35.125104] ---[ end trace 0000000000000000 ]--- [ 35.129801] Kernel panic - not syncing: Fatal exception in interrupt [ 35.136260] CPU3: stopping [ 35.139009] CPU: 3 PID: 27 Comm: migration/3 Tainted: G B D W 5.19.0 #43 [ 35.146872] Hardware name: BCM2711 [ 35.150318] Stopper: multi_cpu_stop+0x0/0x140 <- stop_machine_cpuslocked+0x180/0x1e4 [ 35.158197] unwind_backtrace from show_stack+0x18/0x1c [ 35.163509] show_stack from dump_stack_lvl+0x40/0x4c [ 35.168643] dump_stack_lvl from do_handle_IPI+0x150/0x2a8 [ 35.174218] do_handle_IPI from ipi_handler+0x1c/0x28 [ 35.179351] ipi_handler from handle_percpu_devid_irq+0x94/0x150 [ 35.185454] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48 [ 35.191820] handle_irq_desc from gic_handle_irq+0x6c/0x78 [ 35.197393] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c [ 35.203671] generic_handle_arch_irq from call_with_stack+0x18/0x20 [ 35.210038] call_with_stack from __irq_svc+0x84/0x94 [ 35.215168] Exception stack(0xf0913e98 to 0xf0913ee0) [ 35.220293] 3e80: e7e20a10 00000000 [ 35.228594] 3ea0: 00000000 257dc000 e7e1ec68 f0913ee8 257dc000 00000000 c2806f18 60070013 [ 35.236896] 3ec0: f0863d70 f0863d74 f0863d70 f0913ee8 c02bebd4 c02bebe8 60070013 ffffffff [ 35.245192] __irq_svc from rcu_momentary_dyntick_idle+0x2c/0x9c [ 35.251296] rcu_momentary_dyntick_idle from multi_cpu_stop+0xd4/0x140 [ 35.257931] multi_cpu_stop from cpu_stopper_thread+0x120/0x1d8 [ 35.263947] cpu_stopper_thread from smpboot_thread_fn+0x25c/0x264 [ 35.270228] smpboot_thread_fn from kthread+0x12c/0x140 [ 35.275539] kthread from ret_from_fork+0x14/0x1c [ 35.280317] Exception stack(0xf0913fb0 to 0xf0913ff8) [ 35.285441] 3fa0: 00000000 00000000 00000000 00000000 [ 35.293739] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 35.302037] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 35.308746] CPU2: stopping [ 35.311492] CPU: 2 PID: 22 Comm: migration/2 Tainted: G B D W 5.19.0 #43 [ 35.319355] Hardware name: BCM2711 [ 35.322803] Stopper: multi_cpu_stop+0x0/0x140 <- stop_machine_cpuslocked+0x180/0x1e4 [ 35.330677] unwind_backtrace from show_stack+0x18/0x1c [ 35.335988] show_stack from dump_stack_lvl+0x40/0x4c [ 35.341122] dump_stack_lvl from do_handle_IPI+0x150/0x2a8 [ 35.346697] do_handle_IPI from ipi_handler+0x1c/0x28 [ 35.351830] ipi_handler from handle_percpu_devid_irq+0x94/0x150 [ 35.357932] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48 [ 35.364298] handle_irq_desc from gic_handle_irq+0x6c/0x78 [ 35.369870] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c [ 35.376148] generic_handle_arch_irq from call_with_stack+0x18/0x20 [ 35.382515] call_with_stack from __irq_svc+0x84/0x94 [ 35.387646] Exception stack(0xf08ebea8 to 0xf08ebef0) [ 35.392773] bea0: f0863d70 00000003 00000000 00000001 f0863d60 00000000 [ 35.401074] bec0: 00000001 00000000 c2806f18 600c0013 f0863d70 f0863d74 f0863d70 f08ebef8 [ 35.409372] bee0: c030acac c02bebbc 600c0013 ffffffff [ 35.414495] __irq_svc from rcu_momentary_dyntick_idle+0x0/0x9c [ 35.420511] rcu_momentary_dyntick_idle from 0xc31d0000 [ 35.425820] CPU1: stopping [ 35.428568] CPU: 1 PID: 17 Comm: migration/1 Tainted: G B D W 5.19.0 #43 [ 35.436430] Hardware name: BCM2711 [ 35.439879] Stopper: multi_cpu_stop+0x0/0x140 <- stop_machine_cpuslocked+0x180/0x1e4 [ 35.447752] unwind_backtrace from show_stack+0x18/0x1c [ 35.453064] show_stack from dump_stack_lvl+0x40/0x4c [ 35.458198] dump_stack_lvl from do_handle_IPI+0x150/0x2a8 [ 35.463772] do_handle_IPI from ipi_handler+0x1c/0x28 [ 35.468905] ipi_handler from handle_percpu_devid_irq+0x94/0x150 [ 35.475006] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48 [ 35.481373] handle_irq_desc from gic_handle_irq+0x6c/0x78 [ 35.486945] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c [ 35.493222] generic_handle_arch_irq from call_with_stack+0x18/0x20 [ 35.499590] call_with_stack from __irq_svc+0x84/0x94 [ 35.504721] Exception stack(0xf08c3e98 to 0xf08c3ee0) [ 35.509847] 3e80: e7e00a10 00000000 [ 35.518148] 3ea0: 00000000 257bc000 e7dfec68 f08c3ee8 257bc000 00000000 c2806f18 600f0013 [ 35.526449] 3ec0: f0863d70 f0863d74 f0863d70 f08c3ee8 c02bebd4 c02bebe8 600f0013 ffffffff [ 35.534745] __irq_svc from rcu_momentary_dyntick_idle+0x2c/0x9c [ 35.540849] rcu_momentary_dyntick_idle from multi_cpu_stop+0xd4/0x140 [ 35.547483] multi_cpu_stop from cpu_stopper_thread+0x120/0x1d8 [ 35.553499] cpu_stopper_thread from smpboot_thread_fn+0x25c/0x264 [ 35.559780] smpboot_thread_fn from kthread+0x12c/0x140 [ 35.565090] kthread from ret_from_fork+0x14/0x1c [ 35.569868] Exception stack(0xf08c3fb0 to 0xf08c3ff8) [ 35.574992] 3fa0: 00000000 00000000 00000000 00000000 [ 35.583292] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 35.591589] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 35.599291] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- -- Florian
| |