lkml.org 
[lkml]   [2016]   [Feb]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectPanic in nf_ct_seq_adjust (be2net, synproxy, heavy load)
Date
Hi,

I have finally managed to capture a stack trace from an intermittent
crash we've been observing under heavy load.

There are two identical machines w/dual-port Emulex 10 Gbps NICs running
in an active-passive firewall cluster (though more like passive-passive
in reality). As far as I know, at the time of the reboot there was a
pretty large DoS attack (6 Gbps) on the same network segment, not
directed at the firewalls themselves. The firewalls were being tested
with a ~80Mbps stream of small HTTP requests forwarded with SYNPROXY,
conntrackd and some traffic shaping enabled.

Unfortunately, this trace comes from Debian distro kernel (3.16), as I
haven't yet upgraded this machine and I don't have serial console on the
other one at the moment. The other node is running 4.4.1 and rebooted at
the same time +/- seconds, /proc/uptime says it's up 15 seconds longer.
netconsole stays silent during reboots.

> [436190.637310] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
> [436190.674698] IP: [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack]
> [436190.709683] PGD 0
> [436190.719785] Oops: 0000 [#1] SMP
> [436190.735431] Modules linked in: ipt_SYNPROXY nf_synproxy_core binfmt_misc xt_CLASSIFY xt_mac iptable_filter macvlan ip_set_hash_net cls_u32 sch_red sch_htb nf_conntrac
> k_netlink xt_set ip_set nfnetlink iptable_mangle xt_tcpudp xt_CT xt_conntrack iptable_raw ip_tables arptable_filter arp_tables x_tables intel_powerclamp coretemp kvm_inte
> l kvm crc32_pclmul aesni_intel aes_x86_64 joydev lrw gf128mul glue_helper ttm drm_kms_helper hpwdt drm i2c_algo_bit i2c_core hpilo psmouse evdev iTCO_wdt acpi_power_meter
> pcspkr serio_raw iTCO_vendor_support lpc_ich ablk_helper cryptd shpchp i7core_edac nf_conntrack_ftp mfd_core button edac_core nf_conntrack_ipv4 nf_defrag_ipv4 processor
> nf_conntrack ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct1
> 0dif_pclmul crct10dif_common crc32c_intel be2iscsi ehci_pci uhci_hcd iscsi_boot_sysfs libiscsi ehci_hcd usbcore scsi_transport_iscsi hpsa usb_common be2net scsi_mod therm
> al vxlan thermal_sys
> [436191.160517] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u2
> [436191.208587] Hardware name: HP ProLiant BL460c G7, BIOS I27 07/02/2013
> [436191.239665] task: ffff88020af6c110 ti: ffff88020af78000 task.ti: ffff88020af78000
> [436191.274842] RIP: 0010:[<ffffffffa014af73>] [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack]
> [436191.321186] RSP: 0018:ffff88020ba43c40 EFLAGS: 00010246
> [436191.346218] RAX: 0000000000030003 RBX: ffff880202931300 RCX: 0000000000000003
> [436191.381252] RDX: 00000000ff93df09 RSI: 0000000000000028 RDI: ffff8802064bdb0c
> [436191.414649] RBP: ffff8802064bdb08 R08: ffffffff81459ea0 R09: 0000000000000020
> [436191.449511] R10: ffff880203600000 R11: ffff8800e7b38000 R12: 0000000000000028
> [436191.482769] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801e21b3062
> [436191.516924] FS: 0000000000000000(0000) GS:ffff88020ba40000(0000) knlGS:0000000000000000
> [436191.554343] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [436191.581010] CR2: 0000000000000004 CR3: 0000000001813000 CR4: 00000000000007e0
> [436191.613999] Stack:
> [436191.624137] ffffffff00000014 ffffc9000d8c1800 ffff8802064bdb0c ffffffff00000000
> [436191.660568] ffff8801f92b0af0 0000000000000000 ffff880202931300 ffff8802064bdb08
> [436191.696262] ffff880203600000 0000000000000000 ffff880202931300 ffffffff818e95a0
> [436191.730498] Call Trace:
> [436191.742505] <IRQ>
> [436191.752229] [<ffffffffa0206bd3>] ? ipv4_confirm+0xa3/0xf0 [nf_conntrack_ipv4]
> [436191.786089] [<ffffffff81459ea0>] ? ip_fragment+0x880/0x880
> [436191.813367] [<ffffffff8144f575>] ? nf_iterate+0x65/0xa0
> [436191.838356] [<ffffffff81459ea0>] ? ip_fragment+0x880/0x880
> [436191.864832] [<ffffffff8144f626>] ? nf_hook_slow+0x76/0x130
> [436191.891480] [<ffffffff81459ea0>] ? ip_fragment+0x880/0x880
> [436191.917857] [<ffffffff8145ba42>] ? ip_output+0x82/0x90
> [436191.943454] [<ffffffff8141ebc3>] ? __netif_receive_skb_core+0x543/0x750
> [436191.974406] [<ffffffff8141ee4f>] ? netif_receive_skb_internal+0x1f/0x80
> [436192.006268] [<ffffffff8141f76a>] ? napi_gro_frags+0x1ba/0x2c0
> [436192.032907] [<ffffffffa00dc2f4>] ? be_process_rx+0x2c4/0x780 [be2net]
> [436192.064273] [<ffffffffa00dc9a2>] ? be_poll+0x1f2/0x400 [be2net]
> [436192.091625] [<ffffffff8141f1d0>] ? net_rx_action+0x140/0x240
> [436192.119082] [<ffffffff8106c681>] ? __do_softirq+0xf1/0x290
> [436192.144594] [<ffffffff8106ca55>] ? irq_exit+0x95/0xa0
> [436192.168465] [<ffffffff81516ae2>] ? do_IRQ+0x52/0xe0
> [436192.192353] [<ffffffff8151492d>] ? common_interrupt+0x6d/0x6d
> [436192.219491] <EOI>
> [436192.228773] [<ffffffff8108aded>] ? __hrtimer_start_range_ns+0x1cd/0x390
> [436192.261747] [<ffffffff813dfbbf>] ? cpuidle_enter_state+0x4f/0xc0
> [436192.290427] [<ffffffff813dfbb8>] ? cpuidle_enter_state+0x48/0xc0
> [436192.319838] [<ffffffff810a80d8>] ? cpu_startup_entry+0x2f8/0x400
> [436192.348103] [<ffffffff81042c5f>] ? start_secondary+0x20f/0x2d0
> [436192.376655] Code: 00 00 00 88 44 24 08 83 e0 01 48 8d 14 00 48 01 d0 4d 8d 74 85 00 48 8d 45 04 48 89 c7 48 89 44 24 10 e8 a1 88 3c e1 41 8b 57 04 <41> 8b 76 04 45 8b 46 08 41 8b 7f 08 89 d1 0f c9 41 39 0e 0f 88
> [436192.464643] RIP [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack]
> [436192.499548] RSP <ffff88020ba43c40>
> [436192.515631] CR2: 0000000000000004
> [436192.531367] ---[ end trace 84529a434dcea659 ]---
> [436192.531372] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
> [436192.531380] IP: [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack]
> [436192.531381] PGD 0
> [436192.531382] Oops: 0000 [#2] SMP
> [436192.531402] Modules linked in: ipt_SYNPROXY nf_synproxy_core binfmt_misc xt_CLASSIFY xt_mac iptable_filter macvlan ip_set_hash_net cls_u32 sch_red sch_htb nf_conntrack_netlink xt_set ip_set nfnetlink iptable_mangle xt_tcpudp xt_CT xt_conntrack iptable_raw ip_tables arptable_filter arp_tables x_tables intel_powerclamp coretemp kvm_intel kvm crc32_pclmul aesni_intel aes_x86_64 joydev lrw gf128mul glue_helper ttm drm_kms_helper hpwdt drm i2c_algo_bit i2c_core hpilo psmouse evdev iTCO_wdt acpi_power_meter pcspkr serio_raw iTCO_vendor_support lpc_ich ablk_helper cryptd shpchp i7core_edac nf_conntrack_ftp mfd_core button edac_core nf_conntrack_ipv4 nf_defrag_ipv4 processor nf_conntrack ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common crc32c_intel be2iscsi ehci_pci uhci_hcd iscsi_boot_sysfs libiscsi ehci_hcd usbcore scsi_transport_iscsi hpsa usb_commo
n be2net scsi_mod thermal vxlan thermal_sys
> [436192.531411] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u2
> [436192.531411] Hardware name: HP ProLiant BL460c G7, BIOS I27 07/02/2013
> [436192.531412] task: ffff88020af6d3b0 ti: ffff88020af70000 task.ti: ffff88020af70000
> [436192.531416] RIP: 0010:[<ffffffffa014af73>] [<ffffffffa014af73>] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack]
> [436192.531416] RSP: 0018:ffff88010bc23ca0 EFLAGS: 00010246
> [436192.531417] RAX: 0000000000030003 RBX: ffff8800371f0800 RCX: 0000000000000003
> [436192.531418] RDX: 000000001a710bff RSI: 0000000000000028 RDI: ffff8801f913ac44
> [436192.531418] RBP: ffff8801f913ac40 R08: ffffffff81459ea0 R09: 0000000000000020
> [436192.531419] R10: ffff880203600000 R11: ffff8800e7b38000 R12: 0000000000000028
> [436192.531419] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88010b84b862
> [436192.531420] FS: 0000000000000000(0000) GS:ffff88010bc20000(0000) knlGS:000000

The kernel log ends here, even though it looks like we could be
expecting another identical or similar trace on another CPU (#2 vs #4
earlier).

The issue repeats intermittently but I cannot reliably reproduce it
(except that DoS attacks tend to bring the whole thing down). I don't
know the nature of the attack, but the machines have easily survived 9
Gbps of iperf tests.

Any help is greatly appreciated while I dig into the code on my own.

Best regards,
Grzegorz Nosek

\
 
 \ /
  Last update: 2016-02-16 21:01    [W:0.033 / U:0.168 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site