Messages in this thread | | | Date | Wed, 17 Sep 2008 12:30:32 -0500 | From | Dimitri Sivanich <> | Subject | Re: [RFC 0/4] dynamically allocate arch specific system vectors |
| |
On Tue, Sep 16, 2008 at 03:46:54PM -0500, Dean Nelson wrote: > On Tue, Sep 16, 2008 at 10:24:48AM +0200, Ingo Molnar wrote: > > > > * Dean Nelson <dcn@sgi.com> wrote: > > > > > > while i understand the UV_BAU_MESSAGE case (TLB flushes are > > > > special), why does sgi-gru and sgi-xp need to go that deep? They are > > > > drivers, they should be able to make use of an ordinary irq just > > > > like the other 2000 drivers we have do. > > > > > > The sgi-gru driver needs to be able to allocate a single irq/vector > > > pair for all CPUs even those that are not currently online. The sgi-xp > > > driver has similar but not as stringent needs. > > > > why does it need to allocate a single irq/vector pair? Why is a regular > > interrupt not good? > > When you speak of a 'regular interrupt' I assume you are referring to simply > the irq number, with the knowledge of what vector and CPU(s) it is mapped to > being hidden? > > > sgi-gru driver > > The GRU is not an actual external device that is connected to an IOAPIC. > The gru is a hardware mechanism that is embedded in the node controller > (UV hub) that directly connects to the cpu socket. Any cpu (with permission) > can do direct loads and stores to the gru. Some of these stores will result > in an interrupt being sent back to the cpu that did the store. > > The interrupt vector used for this interrupt is not in an IOAPIC. Instead > it must be loaded into the GRU at boot or driver initialization time. > > The OS needs to route these interrupts back to the GRU driver interrupt > handler on the cpu that received the interrupt. Also, this is a performance > critical path. There should be no globally shared cachelines involved in the > routing. > > The actual vector associated with the IRQ does not matter as long as it is > a relatively high priority interrupt. The vector does need to be mapped to > all of the possible CPUs in the partition. The GRU driver needs to know > vector's value, so that it can load it into the GRU. > > sgi-xp driver > > The sgi-xp driver utilizes the node controller's message queue capability to > send messages from one system partition (a single SSI) to another partition. > > A message queue can be configured to have the node controller raise an > interrupt whenever a message is written into it. This configuration is > accomplished by setting up two processor writable MMRs located in the > node controller. The vector number and apicid of the targeted CPU need > to be written into one of these MMRs. There is no IOAPIC associated with > this. > > So one thought was that, once insmod'd, sgi-xp would allocate a message queue, > allocate an irq/vector pair for a CPU located on the node where the message > queue resides, and then set the MMRs with the memory address and length of the > message queue and the vector and CPU's apicid. And then repeat, as there are > actually two message queues required by this driver.
In addition to the above, the high resolution RTC timers in the UV hardware require that a vector be specified in order to send an interrupt to a specific destination when a timer expires. The MMR's for these timers require a vector to be or'ed in with other values, including the interrupt's destination. This is therefore done at run-time.
Like the GRU's vector, this vector is not in an IOAPIC. This vector would be made available to all cpu's within a partition (SSI) and should be coupled with a per-cpu irq.
This is very similiar to what was available in earlier SGI hardware and used in drivers/char/mmtimer.c.
| |