lkml.org 
[lkml]   [2008]   [Jul]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectBUG on call to pskb_expand_head in skbuff.c
Hi all,
We are using Xen but have been seeing a Linux kernel networking bug
that seems to be outside of the scope of Xen:


Kernel BUG at ...ib/xen-3.2.1/linux-2.6.18-xen.hg/net/core/skbuff.c:695
invalid opcode: 0000 [1] SMP
CPU 3
Modules linked in: xt_mac tun arptable_filter arp_tables xt_physdev
iptable_filter ip_tables x_tables bridge ipv6 button ac battery nfs
lockd nfs_acl sunrpc sg sr_mod parport_pc parport floppy serio_raw
pcspkr i2c_i801 i2c_core joydev ext3 jbd dm_mirror dm_snapshot dm_mod
sd_mod ide_cd cdrom usbhid usb_storage aacraid ehci_hcd e1000 piix
scsi_mod uhci_hcd usbcore thermal processor fan
Pid: 9699, comm: qemu-dm Tainted: GF 2.6.18.8-xen #1
RIP: e030:[<ffffffff803948e5>] [<ffffffff803948e5>]
pskb_expand_head+0x2a/0x138RSP: e02b:ffff880007d0fc08 EFLAGS:
00010202
RAX: 0000000000000001 RBX: ffff880019b6c0c0 RCX: ffff880078365000
RDX: 0000000000000134 RSI: 0000000000000020 RDI: ffff880078365100
RBP: ffff88007f10e000 R08: ffff880078365012 R09: 0000000000000194
R10: ffff88007f10e000 R11: ffffffff8039b349 R12: 0000000000000000
R13: ffff8800789f7e04 R14: 0000000000000002 R15: 0000000000000178
FS: 00002ba5be9835f0(0000) GS:ffffffff804d7180(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process qemu-dm (pid: 9699, threadinfo ffff880048626000, task
ffff880001239080)
Stack: ffff8800789f7dc0 ffff880019b6c0c0 ffff88007f10e000 ffff880019b6c0c0
ffff8800789f7e04 ffffffff80394a4b 0000000000000178 ffff880019b6c0c0
ffff88007f10e000 ffff880019b6c0c0 ffff8800789f7e04 0000000000000002
Call Trace:
<IRQ> [<ffffffff80394a4b>] __pskb_pull_tail+0x58/0x26e
[<ffffffff8039b4a3>] dev_queue_xmit+0x15a/0x313
[<ffffffff8039e9e2>] neigh_update+0x304/0x3d9
[<ffffffff803a7ccd>] eth_header_cache_update+0x0/0x12
[<ffffffff803d6cc8>] arp_process+0x579/0x5c2
[<ffffffff803d674f>] arp_process+0x0/0x5c2
[<ffffffff803b0b22>] nf_hook_slow+0x58/0xc4
[<ffffffff803d674f>] arp_process+0x0/0x5c2
[<ffffffff803d6e17>] arp_rcv+0x106/0x129
[<ffffffff80398d95>] netif_receive_skb+0x0/0x2eb
[<ffffffff80398ffb>] netif_receive_skb+0x266/0x2eb
[<ffffffff88247950>] :bridge:br_pass_frame_up+0x67/0x69
[<ffffffff88247a18>] :bridge:br_handle_frame_finish+0xc6/0xf8
[<ffffffff88247bd2>] :bridge:br_handle_frame+0x188/0x1a6
[<ffffffff80398f5f>] netif_receive_skb+0x1ca/0x2eb
[<ffffffff8039af67>] process_backlog+0xd0/0x182
[<ffffffff8039b1e2>] net_rx_action+0xe3/0x24a
[<ffffffff802356ec>] __do_softirq+0x83/0x117
[<ffffffff8020b1ac>] call_softirq+0x1c/0x28
<EOI> [<ffffffff8020d01f>] do_softirq+0x6a/0xeb
[<ffffffff8039936e>] netif_rx_ni+0x19/0x1d
[<ffffffff882702b8>] :tun:tun_chr_writev+0x1d0/0x204
[<ffffffff88270306>] :tun:tun_chr_write+0x1a/0x1f
[<ffffffff802803f7>] vfs_write+0xce/0x174
[<ffffffff802809b5>] sys_write+0x45/0x6e
[<ffffffff8020a4fc>] system_call+0x68/0x6d
[<ffffffff8020a494>] system_call+0x0/0x6d

I've already posted this to xen-devel and Keir Fraser has already told
me that this is unrelated to any Xen modifications so I thought I'd try
here. We have seen this same traceback on two identical hardware systems
with the following configurations

- Supermicro X7DBE+ Motherboard
- Integrated Intel Corporation 631xESB/632xESB DPT NIC

The host systems are running Debian 4.0 Etch amd64 with a stock Xen
kernel pulled from http://xenbits.xensource.com/ and compiled with the
default configuration. Here is lspci output

00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller
Hub (rev b1)
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8
Port 2-3 (rev b1)
00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8
Port 4-5 (rev b1)
00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8
Port 6-7 (rev b1)
00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA
Engine (rev b1)
00:10.0 Host bridge: Intel Corporation 5000 Series Chipset Error
Reporting Registers (rev b1)
00:10.1 Host bridge: Intel Corporation 5000 Series Chipset Error
Reporting Registers (rev b1)
00:10.2 Host bridge: Intel Corporation 5000 Series Chipset Error
Reporting Registers (rev b1)
00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
Registers (rev b1)
00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
Registers (rev b1)
00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers
(rev b1)
00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers
(rev b1)
00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI
Express Root Port 1 (rev 09)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #1 (rev 09)
00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #2 (rev 09)
00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #3 (rev 09)
00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC
Interface Controller (rev 09)
00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller
(rev 09)
00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus
Controller (rev 09)
01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
Upstream Port (rev 01)
01:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to
PCI-X Bridge (rev 01)
02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
Downstream Port E1 (rev 01)
02:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
Downstream Port E3 (rev 01)
03:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge
A (rev 09)
03:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge
B (rev 09)
06:00.0 Ethernet controller: Intel Corporation 631xESB/632xESB DPT LAN
Controller Copper (rev 01)
06:00.1 Ethernet controller: Intel Corporation 631xESB/632xESB DPT LAN
Controller Copper (rev 01)
09:00.0 PCI bridge: Intel Corporation 80333 Segment-A PCI Express-to-PCI
Express Bridge
09:00.2 PCI bridge: Intel Corporation 80333 Segment-B PCI Express-to-PCI
Express Bridge
0a:0e.0 RAID bus controller: Adaptec AAC-RAID
0d:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)


And kernel version

Linux version 2.6.18.8-xen (xxxx) (gcc version 4.1.2 20061115
(prerelease) (Debian 4.1.1-21)) #1 SMP Wed Apr 30 14:27:56 EDT 2008

The system is crashing at line 695 in core/net/skbuff.c (Xen source tree
anyway). After getting a couple of these crashes we replaced the code in
skbuff.c in pskb_expand_head

if (skb_shared(skb))
BUG();

to instead read

if (skb_shared(skb)) {
printk(KERN_INFO "skb_shared BUG problem detected\n");
printk(KERN_INFO " skb->users: %d\n", atomic_read(&skb->users));
printk(KERN_INFO " skb->dev: %s\n", skb->dev->name);
printk(KERN_INFO " skb->fclone: %d\n", skb->fclone);
}

We also went ahead and cleared out all iptables and arptables rules,
made sure that iptables and arptables modules wouldn't load again and
restarted the system. After doing this the bug was triggered within a
day while under normal load and network activity. In dmesg output we got
this from the debugging code above

skb_shared BUG problem detected
skb->users: 2
skb->dev: br0
skb->fclone: 0

My initial suspicion was that it was somehow related to arptables given
the initial traceback but given that it was reproduced without iptables
and arptables modules loaded makes me now think otherwise. The network
setup for this system consists of Xen guests with their interfaces all
bridged on br0. IP forwarding is enabled and traffic is routed between
the bridge device and the external network on eth1.

At the moment we have not been able to trigger this bug at will so we
don't have any way to test this on a newer kernel. If there is any more
information that you think I may be able to provide I would be happy to
provide all and any information possible. Here is the thread on
xen-devel for reference

http://article.gmane.org/gmane.comp.emulators.xen.devel/52881

If anyone has any ideas or suggestions I'd really appreciate insight
into this issue.

Much Thanks,
Christopher Thunes




\
 
 \ /
  Last update: 2008-07-25 21:11    [W:0.062 / U:0.768 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site