lkml.org 
[lkml]   [2006]   [Apr]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: irqbalance mandatory on SMP kernels?
    Erik Mouw wrote:
    > On Tue, Apr 18, 2006 at 08:19:17PM +0200, Arjan van de Ven wrote:
    >
    >>>but spreading IRQ's across all of the CPU's doesn't seem like it's
    >>>ever the right answer.
    >>
    >>well it is in some cases, imagine having 2 cpus and 2 gige nics that are
    >>very busy doing webserving. That's an obvious case where 1-nic-per-cpu
    >>ends up doing the right thing... the way it ends up is that each nic has
    >>a full cpu for itself and it's own apaches... almost fully independent
    >>of the other one. Now if you moved both irqs to the same cpu, the
    >>apaches would follow, because if they didn't then you'd be bouncing
    >>their data *all the time*. And at that point the other cpu will become
    >>bored ;)
    >
    >
    > So what happens with a dual amd64 system where each CPU has its "own"
    > NIC? Something like this:
    >
    >
    > MEM0 <--> CPU0 <--- HT ---> CPU1 <--> MEM1
    > ^ ^
    > | |
    > HT HT
    > | |
    > v v
    > PCI bridge0 PCI1 bridge
    > ^ ^
    > | |
    > PCI PCI
    > | |
    > v v
    > GigE NIC0 GigE NIC1
    >
    > The "best" approach would be to run an Apache on each CPU using local
    > memory and NIC and having the IRQs handled by the local CPU. Does the
    > irq balancer allow such a configuraion, or would it be hamperd by the
    > process scheduler deciding to run both Apaches on a single CPU?

    The trouble is that we're not smart enough to redirect receiving traffic
    back to the correct CPU, I think. You'd need to configure different IP
    addrs on the same subnet to each NIC, and have intelligent bonding for
    outbound traffic.

    Then you'd need NUMA locality of IRQs, by bonding them to their closest
    CPUs, which is something that's easily done inside the kernel, but is
    harder in userspace. but I'm not getting into that silly pissing contest
    again. Either mechanism *could* do it. It's just about creating sensbile
    APIs and defaults, both of which we're not good at doing as a community.

    M.
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2006-04-19 16:33    [W:0.023 / U:95.128 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site