lkml.org 
[lkml]   [2008]   [Jan]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    Subjectquestions on NAPI processing latency and dropped network packets
    Hi all,

    I've got an issue that's popped up with a deployed system running
    2.6.10. I'm looking for some help figuring out why incoming network
    packets aren't being processed fast enough.

    After a recent userspace app change, we've started seeing packets being
    dropped by the ethernet hardware (e1000, NAPI is enabled). The
    error/dropped/fifo counts are going up in ethtool:

    rx_packets: 32180834
    rx_bytes: 5480756958
    rx_errors: 862506
    rx_dropped: 771345
    rx_length_errors: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_fifo_errors: 91161
    rx_missed_errors: 91161

    This link is receiving roughly 13K packets/sec, and we're dropping
    roughly 51 packets/sec due to fifo errors.

    Increasing the rx descriptor ring size from 256 up to around 3000 or so
    seems to make the problem stop, but it seems to me that this is just a
    workaround for the latency in processing the incoming packets.

    So, I'm looking for some suggestions on how to fix this or to figure out
    where the latency is coming from.

    Some additional information:


    1) Interrupts are being processed on both cpus:

    root@base0-0-0-13-0-11-1:/root> cat /proc/interrupts
    CPU0 CPU1
    30: 1703756 4530785 U3-MPIC Level eth0




    2) "top" shows a fair amount of time processing softirqs, but very
    little time in ksoftirqd (or is that a sampling artifact?).


    Tasks: 79 total, 1 running, 78 sleeping, 0 stopped, 0 zombie
    Cpu0: 23.6% us, 30.9% sy, 0.0% ni, 36.9% id, 0.0% wa, 0.3% hi, 8.3% si
    Cpu1: 30.4% us, 24.1% sy, 0.0% ni, 5.9% id, 0.0% wa, 0.7% hi, 38.9% si
    Mem: 4007812k total, 2199148k used, 1808664k free, 0k buffers
    Swap: 0k total, 0k used, 0k free, 219844k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    5375 root 15 0 2682m 1.8g 6640 S 99.9 46.7 31:17.68
    SigtranServices
    7696 root 17 0 6952 3212 1192 S 7.3 0.1 0:15.75
    schedmon.ppc210
    7859 root 16 0 2688 1228 964 R 0.7 0.0 0:00.04 top
    2956 root 8 -8 18940 7436 5776 S 0.3 0.2 0:01.35 blademtc
    1 root 16 0 1660 620 532 S 0.0 0.0 0:30.62 init
    2 root RT 0 0 0 0 S 0.0 0.0 0:00.01 migration/0
    3 root 15 0 0 0 0 S 0.0 0.0 0:00.55 ksoftirqd/0
    4 root RT 0 0 0 0 S 0.0 0.0 0:00.01 migration/1
    5 root 15 0 0 0 0 S 0.0 0.0 0:00.43 ksoftirqd/1


    3) /proc/sys/net/core/netdev_max_backlog is set to the default of 300


    So...anyone have any ideas/suggestions?

    Thanks,

    Chris


    \
     
     \ /
      Last update: 2008-01-10 18:27    [W:0.022 / U:34.188 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site