lkml.org 
[lkml]   [2013]   [Jun]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: Scaling problem with a lot of AF_PACKET sockets on different interfaces
From
Date
(CC's net-fu dojo) 

On Fri, 2013-06-07 at 14:56 +0300, Vitaly V. Bursov wrote:
> Hello,
>
> I have a Linux router with a lot of interfaces (hundreds or
> thousands of VLANs) and an application that creates AF_PACKET
> socket per interface and bind()s sockets to interfaces.
>
> Each socket has attached BPF filter too.
>
> The problem is observed on linux-3.8.13, but as far I can see
> from the source the latest version has alike behavior.
>
> I noticed that box has strange performance problems with
> most of the CPU time spent in __netif_receive_skb:
> 86.15% [k] __netif_receive_skb
> 1.41% [k] _raw_spin_lock
> 1.09% [k] fib_table_lookup
> 0.99% [k] local_bh_enable_ip
>
> and this the assembly with the "hot spot":
> │ shr $0x8,%r15w
> │ and $0xf,%r15d
> 0.00 │ shl $0x4,%r15
> │ add $0xffffffff8165ec80,%r15
> │ mov (%r15),%rax
> 0.09 │ mov %rax,0x28(%rsp)
> │ mov 0x28(%rsp),%rbp
> 0.01 │ sub $0x28,%rbp
> │ jmp 5c7
> 1.72 │5b0: mov 0x28(%rbp),%rax
> 0.05 │ mov 0x18(%rsp),%rbx
> 0.00 │ mov %rax,0x28(%rsp)
> 0.03 │ mov 0x28(%rsp),%rbp
> 5.67 │ sub $0x28,%rbp
> 1.71 │5c7: lea 0x28(%rbp),%rax
> 1.73 │ cmp %r15,%rax
> │ je 640
> 1.74 │ cmp %r14w,0x0(%rbp)
> │ jne 5b0
> 81.36 │ mov 0x8(%rbp),%rax
> 2.74 │ cmp %rax,%r8
> │ je 5eb
> 1.37 │ cmp 0x20(%rbx),%rax
> │ je 5eb
> 1.39 │ cmp %r13,%rax
> │ jne 5b0
> 0.04 │5eb: test %r12,%r12
> 0.04 │ je 6f4
> │ mov 0xc0(%rbx),%eax
> │ mov 0xc8(%rbx),%rdx
> │ testb $0x8,0x1(%rdx,%rax,1)
> │ jne 6d5
>
> This corresponds to:
>
> net/core/dev.c:
> type = skb->protocol;
> list_for_each_entry_rcu(ptype,
> &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {
> if (ptype->type == type &&
> (ptype->dev == null_or_dev || ptype->dev == skb->dev ||
> ptype->dev == orig_dev)) {
> if (pt_prev)
> ret = deliver_skb(skb, pt_prev, orig_dev);
> pt_prev = ptype;
> }
> }
>
> Which works perfectly OK until there are a lot of AF_PACKET sockets, since
> the socket adds a protocol to ptype list:
>
> # cat /proc/net/ptype
> Type Device Function
> 0800 eth2.1989 packet_rcv+0x0/0x400
> 0800 eth2.1987 packet_rcv+0x0/0x400
> 0800 eth2.1986 packet_rcv+0x0/0x400
> 0800 eth2.1990 packet_rcv+0x0/0x400
> 0800 eth2.1995 packet_rcv+0x0/0x400
> 0800 eth2.1997 packet_rcv+0x0/0x400
> .......
> 0800 eth2.1004 packet_rcv+0x0/0x400
> 0800 ip_rcv+0x0/0x310
> 0011 llc_rcv+0x0/0x3a0
> 0004 llc_rcv+0x0/0x3a0
> 0806 arp_rcv+0x0/0x150
>
> And this obviously results in a huge performance penalty.
>
> ptype_all, by the looks, should be the same.
>
> Probably one way to fix this it to perform interface name matching in
> af_packet handler, but there could be other cases, other protocols.
>
> Ideas are welcome :)
>


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2013-06-07 15:21    [W:0.100 / U:0.024 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site