`On Wed, Sep 06, 2017 at 12:57:41AM +0200, Thomas Gleixner wrote:> On Sun, 3 Sep 2017, Thomas Gleixner wrote:> > > On Fri, 1 Sep 2017, Chen Yu wrote:> > > > > This is the major logic to spread the vectors on different CPUs.> > > The main idea is to choose the 'idlest' CPU which has assigned> > > the least number of vectors as the candidate/hint for the vector> > > allocation domain, in the hope that the vector allocation domain> > > could leverage this hint to generate corresponding cpumask.> > > > > > One of the requirements to do this vector spreading work comes from the> > > following hibernation problem found on a 16 cores server:> > > > > > CPU 31 disable failed: CPU has 62 vectors assigned and there> > > are only 0 available.> > Thinking more about this, this makes no sense whatsoever.> > The total number of interrupts on a system is the same whether they are> all on CPU 0 or evenly spread over all CPUs.> > As this machine is using physcial destination mode, the number of vectors> used is the same as the number of interrupts, except for the case where a> move of an interrupt is in progress and the interrupt which cleans up the> old vector has not yet arrived. Lets ignore that for now.> > The available vector space is 204 per CPU on such a system.> >     256 - SYSTEM[0-31, 32, 128, 239-255] - LEGACY[50] = 204> > > > CPU 31 disable failed: CPU has 62 vectors assigned and there> > > are only 0 available.> > CPU31 is the last AP going offline (CPU0 is still online).> > It wants to move 62 vectors to CPU0, but it can't because CPU0 has 0> available vectors. That means CPU0 has 204 vectors used. I doubt that, but> what I doubt even more is that this interrupt spreading helps in any way.> > Assumed that we have a total of 204 + 62 = 266 device interrupt vectors in> use and they are evenly spread over 32 CPUs, so each CPU has either 8 or> nine vectors. Fine.> > Now if you unplug all CPUs except CPU0 starting from CPU1 up to CPU31 then> at the point where CPU31 is about to be unplugged, CPU0 holds 133 vectors> and CPU31 holds 133 vectors as well - assumed that the spread is exactly> even.> > I have a hard time to figure out how the 133 vectors on CPU31 are now> magically fitting in the empty space on CPU0, which is 204 - 133 = 71. In> my limited understanding of math 133 is greater than 71, but your patch> might make that magically be wrong.>The problem is reproduced when the network cable is not plugged in,because this driver looks like this:step 1. Reserved enough irq vectors and corresponding IRQs.step 2. If the network is activated, invoke request_irq() to        register the handler.step 3. Invoke set_affinity() to spread the IRQs onto different        CPUs, thus to spread the vectors too.Here's my understanding for why spreading vectors might help for thisspecial case: As step 2 will not get invoked, the IRQs of this driverhas not been enabled, thus in migrate_one_irq() this IRQwill not be considered because there is a check ofirqd_is_started(d), thus there should only be 8 vectorsallocated by this driver on CPU0, and 8 vectors left onCPU31, and the 8 vectors on CPU31 will not be migratedto CPU0 neither, so there is room for other 'valid' vectorsto be migrated to CPU0.> Can you please provide detailed information about how many device> interrupts are actually in use/allocated on that system?> > Please enable CONFIG_GENERIC_IRQ_DEBUGFS and provide the output of > > # cat /sys/kernel/debug/irq/domains/*>> and> > # ls /sys/kernel/debug/irq/irqs>Ok, here's the information after system bootup on topof 4.13:# cat /sys/kernel/debug/irq/domains/*name:   VECTOR size:   0 mapped: 388 flags:  0x00000041name:   IO-APIC-0 size:   24 mapped: 16 flags:  0x00000041 parent: VECTOR    name:   VECTOR     size:   0     mapped: 388     flags:  0x00000041name:   IO-APIC-1 size:   8 mapped: 2 flags:  0x00000041 parent: VECTOR    name:   VECTOR     size:   0     mapped: 388     flags:  0x00000041name:   IO-APIC-2 size:   8 mapped: 0 flags:  0x00000041 parent: VECTOR    name:   VECTOR     size:   0     mapped: 388     flags:  0x00000041name:   IO-APIC-3 size:   8 mapped: 0 flags:  0x00000041 parent: VECTOR    name:   VECTOR     size:   0     mapped: 388     flags:  0x00000041name:   IO-APIC-4 size:   8 mapped: 5 flags:  0x00000041 parent: VECTOR    name:   VECTOR     size:   0     mapped: 388     flags:  0x00000041name:   PCI-HT size:   0 mapped: 0 flags:  0x00000041 parent: VECTOR    name:   VECTOR     size:   0     mapped: 388     flags:  0x00000041name:   PCI-MSI-2 size:   0 mapped: 365 flags:  0x00000051 parent: VECTOR    name:   VECTOR     size:   0     mapped: 388     flags:  0x00000041name:   VECTOR size:   0 mapped: 388 flags:  0x00000041# ls /sys/kernel/debug/irq/irqsls /sys/kernel/debug/irq/irqs0  10   11  13  142  184  217  259  292  31  33   337  339340  342  344  346  348  350  352  354  356  358  360  362364  366  368  370  372  374  376  378  380  382  384  386388  390  392  394  4  6   7  9  1  109  12  14  15   224   26   3    32  335  338  34   341  343  345  347  349351  353  355  357  359  361  363  365  367  369  371  373375  377  379  381  383  385  387  389  391  393  395  567  8BTW, do we have sysfs to display how much vectors used on each CPUs?Thanks,	Yu> Thanks,> > 	tglx> > > > > >    > > > > > `