Messages in this thread | | | From | Ian Kumlien <> | Date | Mon, 26 Jun 2023 19:30:22 +0200 | Subject | Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading? |
| |
There, that didn't take long, even with wireguard disabled
[14079.678380] BUG: kernel NULL pointer dereference, address: 00000000000000c0 [14079.685456] #PF: supervisor read access in kernel mode [14079.690686] #PF: error_code(0x0000) - not-present page [14079.695915] PGD 0 P4D 0 [14079.698540] Oops: 0000 [#1] PREEMPT SMP NOPTI [14079.702996] CPU: 11 PID: 891 Comm: napi/eno2-80 Not tainted 6.4.0 #360 [14079.709614] Hardware name: Supermicro Super Server/A2SDi-12C-HLN4F, BIOS 1.7a 10/13/2022 [14079.717796] RIP: 0010:__udp_gso_segment+0x346/0x4f0 [14079.722778] Code: c3 08 66 89 5c 02 04 45 84 e4 0f 85 27 fd ff ff 49 8b 1e 49 8b ae c0 00 00 00 41 0f b7 86 b4 00 00 00 45 0f b7 a6 b2 00 00 00 <48> 8b b3 c0 00 00 00 0f b7 8b b2 00 00 00 49 01 ec 48 01 c5 48 8d [14079.741645] RSP: 0018:ffffa83643a4f818 EFLAGS: 00010246 [14079.746966] RAX: 00000000000000ce RBX: 0000000000000000 RCX: 0000000000000000 [14079.754195] RDX: ffffa2ad1403b000 RSI: 0000000000000028 RDI: ffffa2afc9d302d4 [14079.761422] RBP: ffffa2ad1403b000 R08: 0000000000000022 R09: 00002000001558c9 [14079.768650] R10: 0000000000000000 R11: ffffa2b02fcea888 R12: 00000000000000e2 [14079.775879] R13: ffffa2afc9d30200 R14: ffffa2afc9d30200 R15: 00002000001558c9 [14079.783106] FS: 0000000000000000(0000) GS:ffffa2b02fcc0000(0000) knlGS:0000000000000000 [14079.791305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14079.797162] CR2: 00000000000000c0 CR3: 0000000151ff4000 CR4: 00000000003526e0 [14079.804408] Call Trace: [14079.806961] <TASK> [14079.809170] ? __die+0x1a/0x60 [14079.812340] ? page_fault_oops+0x158/0x440 [14079.816551] ? ip6_route_output_flags+0xe3/0x160 [14079.821284] ? exc_page_fault+0x3f4/0x820 [14079.825408] ? update_load_avg+0x77/0x710 [14079.829534] ? asm_exc_page_fault+0x22/0x30 [14079.833836] ? __udp_gso_segment+0x346/0x4f0 [14079.838218] ? __udp_gso_segment+0x2fa/0x4f0 [14079.842600] ? _raw_spin_unlock_irqrestore+0x16/0x30 [14079.847679] ? try_to_wake_up+0x8e/0x5a0 [14079.851713] inet_gso_segment+0x150/0x3c0 [14079.855827] ? vhost_poll_wakeup+0x31/0x40 [14079.860032] skb_mac_gso_segment+0x9b/0x110 [14079.864331] __skb_gso_segment+0xae/0x160 [14079.868455] ? netif_skb_features+0x144/0x290 [14079.872928] validate_xmit_skb+0x167/0x370 [14079.877139] validate_xmit_skb_list+0x43/0x70 [14079.881612] sch_direct_xmit+0x267/0x380 [14079.885641] __qdisc_run+0x140/0x590 [14079.889324] __dev_queue_xmit+0x44d/0xba0 [14079.893450] ? nf_hook_slow+0x3c/0xb0 [14079.897229] br_dev_queue_push_xmit+0xb2/0x1c0 [14079.901788] maybe_deliver+0xa9/0x100 [14079.905564] br_flood+0x8a/0x180 [14079.908903] br_handle_frame_finish+0x31f/0x5b0 [14079.913547] br_handle_frame+0x28f/0x3a0 [14079.917585] ? ipv6_find_hdr+0x1f0/0x3e0 [14079.921622] ? br_handle_local_finish+0x20/0x20 [14079.926267] __netif_receive_skb_core.constprop.0+0x4c5/0xc90 [14079.932125] ? br_handle_frame_finish+0x5b0/0x5b0 [14079.936946] ? ___slab_alloc+0x4bf/0xaf0 [14079.940986] __netif_receive_skb_list_core+0x107/0x250 [14079.946240] netif_receive_skb_list_internal+0x194/0x2b0 [14079.951660] ? napi_gro_flush+0x97/0xf0 [14079.955604] napi_complete_done+0x69/0x180 [14079.959808] ixgbe_poll+0xe10/0x12e0 [14079.963506] __napi_poll+0x26/0x1b0 [14079.967106] napi_threaded_poll+0x232/0x250 [14079.971405] ? __napi_poll+0x1b0/0x1b0 [14079.975260] kthread+0xee/0x120 [14079.978510] ? kthread_complete_and_exit+0x20/0x20 [14079.983415] ret_from_fork+0x22/0x30 [14079.987102] </TASK> [14079.989395] Modules linked in: chaoskey [14079.993347] CR2: 00000000000000c0 [14079.996773] ---[ end trace 0000000000000000 ]--- [14080.018013] pstore: backend (erst) writing error (-28) [14080.023274] RIP: 0010:__udp_gso_segment+0x346/0x4f0 [14080.028264] Code: c3 08 66 89 5c 02 04 45 84 e4 0f 85 27 fd ff ff 49 8b 1e 49 8b ae c0 00 00 00 41 0f b7 86 b4 00 00 00 45 0f b7 a6 b2 00 00 00 <48> 8b b3 c0 00 00 00 0f b7 8b b2 00 00 00 49 01 ec 48 01 c5 48 8d [14080.047181] RSP: 0018:ffffa83643a4f818 EFLAGS: 00010246 [14080.052522] RAX: 00000000000000ce RBX: 0000000000000000 RCX: 0000000000000000 [14080.059765] RDX: ffffa2ad1403b000 RSI: 0000000000000028 RDI: ffffa2afc9d302d4 [14080.067012] RBP: ffffa2ad1403b000 R08: 0000000000000022 R09: 00002000001558c9 [14080.074257] R10: 0000000000000000 R11: ffffa2b02fcea888 R12: 00000000000000e2 [14080.081502] R13: ffffa2afc9d30200 R14: ffffa2afc9d30200 R15: 00002000001558c9 [14080.088746] FS: 0000000000000000(0000) GS:ffffa2b02fcc0000(0000) knlGS:0000000000000000 [14080.096964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14080.102823] CR2: 00000000000000c0 CR3: 0000000151ff4000 CR4: 00000000003526e0 [14080.110067] Kernel panic - not syncing: Fatal exception in interrupt [14080.325501] Kernel Offset: 0x12600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [14080.353129] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
On Mon, Jun 26, 2023 at 7:24 PM Ian Kumlien <ian.kumlien@gmail.com> wrote: > > On Mon, Jun 26, 2023 at 7:15 PM Alexander Lobakin > <aleksander.lobakin@intel.com> wrote: > > > > From: Ian Kumlien <ian.kumlien@gmail.com> > > Date: Mon, 26 Jun 2023 16:25:24 +0200 > > > > > On Mon, Jun 26, 2023 at 4:18 PM Alexander Lobakin > > > <aleksander.lobakin@intel.com> wrote: > > >> > > >> From: Ian Kumlien <ian.kumlien@gmail.com> > > >> Date: Sun, 25 Jun 2023 12:59:54 +0200 > > >> > > >>> It could actually be that it's related to: rx-gro-list but > > >>> rx-udp-gro-forwarding makes it trigger quicker... I have yet to > > >>> trigger it on igb > > >> > > >> Hi, the rx-udp-gro-forwarding author here. > > >> > > >> (good thing this appeared on IWL, which I read time to time, but please > > >> Cc netdev next time) > > >> (thus +Cc Jakub, Eric, and netdev) > > > > > > Well, two things, it seems like rx-udp-gro-forwarding accelerates it > > > but the issue is actually in: rx-gro-list > > > > Do you enable them simultaneously? I remember, when I was adding > > gro-fwd, it was working (and working good) as follows: > > > > 1. gro-fwd on, gro-list off: gro-fwd > > 2. gro-fwd off, gro-list on: gro-list > > 3. gro-fwd on, gro-list on: gro-list > > > > Note that their receive paths are independent[0]: skb_gro_receive_list() > > vs skb_gro_receive(), thus I'm still not really sure how gro-fwd can > > trigger gro-list's bug. > > Neither am I... I have enabled sol via ipmitool now, will try to get a > better capture > > > > And since i've only been able to trigger it in ixgbe i thought it > > > might be a driver issue =) > > > > Your screenshot says "__udp_gso_segment", which means that the > > problematic UDP GRO packet hits the Tx path. Rx is in general > > driver-independent. Tx has separate netdev feature ("tx-gso-list"), but > > it's not supported by any driver, just software stack. It might be that > > your traffic goes through a bridge or tunnel or anything else that > > triggers GSO and software segmentation then booms for some reason. > > BTW, __udp_gso_segment() is one-liner when the passed skb was > > gro-listed[1], so having it in the bug splat could mean the skb didn't > > take that route. But hard to say with no full stacktrace. > > I do have a UDP tunnel, in wireguard, will disable it. > > Beyond that some bridges and veth interfaces, but lets wait for a full trace > > > [...] > > > > >>>> But correlating that with the source is beyond me, it could be generic > > >>>> but i thought i'd send it you first since it's part of the redhat > > >>>> guide to speeding up udp traffic > > >> [0] > > >> https://lore.kernel.org/netdev/f83d79d6-f8d7-a229-941a-7d7427975160@nvidia.com > > >> > > >> Thanks, > > >> Olek > > > > [0] > > https://elixir.bootlin.com/linux/latest/source/net/ipv4/udp_offload.c#L518 > > [1] > > https://elixir.bootlin.com/linux/latest/source/net/ipv4/udp_offload.c#L277 > > > > Thanks, > > Olek
| |