[lkml]   [2001]   [Feb]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: New net features for added performance
    Jeff Garzik wrote:
    > 1) Rx Skb recycling.
    > 2) Tx packet grouping.
    > 3) Slabbier packet allocation.

    Let's see what the profiler says. 10 seconds of TCP xmit
    followed by 10 seconds of TCP receive. 100 mbits/sec.
    Kernel 2.4.2+ZC.

    c0119470 do_softirq 97 0.7132
    c020e718 ip_output 99 0.3694
    c020a2c8 ip_route_input 103 0.2893
    c01fdc4c skb_release_data 113 1.0089
    c021312c tcp_sendmsg 113 0.0252
    c0129c64 kmalloc 117 0.3953
    c0112efc __wake_up_sync 128 0.6667
    c01fdd24 __kfree_skb 153 0.6071
    c020e824 ip_queue_xmit 154 0.1149
    c011be80 del_timer 163 2.2639
    c0222fac tcp_v4_rcv 173 0.1022
    c010a778 handle_IRQ_event 178 1.4833
    c01127fc schedule 200 0.1259
    c01d39f8 boomerang_rx 332 0.2823
    c024284c csum_partial_copy_generic 564 2.2742
    c01d2c84 boomerang_start_xmit 654 0.9033
    c0242b3c __generic_copy_from_user 733 12.2167
    c01d329c boomerang_interrupt 910 0.8818
    c01071f4 poll_idle 41813 1306.6562
    00000000 total 48901 0.0367

    7088 non-idle ticks.
    153+117+113 ticks in skb/memory type functions.

    So, naively, the most which can be saved here by optimising
    the skb and memory usage is 5% of networking load. (1% of
    system load @100 mbps)

    Total device driver cost is 27% of the networking load.

    All the meat is in the interrupt load. The 3com driver
    transfers about three packets per interrupt. Here's
    the system load (dual CPU):

    Doing 100mbps TCP send with netperf: 14.9%
    Doing 100mbps TCP receive with netperf: 23.3%

    When tx interrupt mitigation is disabled we get 1.5 packets
    per interrupt doing transmit:

    Doing 100mbps TCP send with netperf: 16.1%
    Doing 100mbps TCP receive with netperf: 24.0%

    So a 2x reduction in interrupt frequency on TCP transmit has
    saved 1.2% of system load. That's 8% of networking load, and,
    presumably, 30% of the driver load. That all seems to make sense.

    The moral?

    - Tuning skb allocation isn't likely to make much difference.
    - At the device-driver level the most effective thing is
    to reduce the number of interrupts.
    - If we can reduce the driver cost to *zero*, we improve
    TCP efficiency by 27%.
    - At the system level the most important thing is to rewrite
    applications to use sendfile(). (But Rx is more expensive
    than Tx, so even this ain't the main game).

    I agree that batching skbs into hard_start_xmit() may allow
    some driver optimisations. Pass it a vector of skbs rather
    than one, and let it return an indication of how many were
    actually consumed. But we'd need to go through an exercise
    like the above beforehand - it may not be worth the
    protocol-level trauma.

    I suspect that a thorough analysis of the best way to
    use Linux networking, and then a rewrite of important
    applications so they use the result of that analysis
    would pay dividends.

    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2005-03-22 13:17    [W:0.024 / U:4.760 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site