Messages in this thread |  | | | Subject | [PATCH][2.4] generic cluster APIC support for systems with more than 8 CPUs | | Date | Wed, 18 Dec 2002 14:36:20 -0800 | | From | "Pallipadi, Venkatesh" <> |
| |
2.4.21-pre1 (i386) based patch to fix the issues with systems having more than 8 CPUs, in a generic way.
Motivation: The current APIC destination mode ("Flat Logical") used in linux kernel has an upper limit of 8 CPUs. For more than 8 CPUs, either "Clustered Logical" or "Physical" mode has to be used. There is already some code in current kernel (2.4.21-pre1), to support such conditions. Specifically, IBM Summit, uses Physical mode, and IBM NUMAQ uses clustered mode. But, there are some issues too: - it is not generic enough to support any other more than 8 CPU system out of the box. Supporting different systems may need more hacks in the code. - clustered mode setup is tightly coupled with NUMA system. Whereas, in reality, we can as well have logical clusters in a non-NUMA system as well. - physical mode setup is somewhat tightly coupled with xAPIC. But, xAPIC doesn't necessarily imply physical mode. You can as well have clustered mode with xAPIC - APIC destination mode is selected based on particular OEM string.
These reasons together led to panics on other OEM systems with > 8 CPUS. The patch tries to fix this issue in a generic way (in place of having multiple hacks for different OEMs). Note, the patch only intends to change the initialization of systems with more than 8 CPUs and it will not affect other systems (apart from possible bugs in my code itself).
Description: The basic idea is to use the number of processors detected on the system, to decide on which APIC destination mode is to be used. Once all the CPU info, is collected either from ACPI or MP table, we can check the total number of processors in the system. If the number of processors in less than equal to 8, then no change is required, and we can use the default, "Flat Logical" set up. If the number of processors is more than 8 we can switch to clustered logical setup. The logical clusters set up as follows. Cluster 0 (CPU 0-3), Cluster 1 (CPU 4-7), Cluster 2 (CPU 8-11) and so on..
The other things that are done in the patch include: - Separate out the NUMA specific stuff from APIC setup in cluster mode. Also, NUMA has its own way of setting up the clusters, and doesn't follow the logical cluster mapping defined above. - Separate out xAPIC stuff from APIC destination setup. And the availability of xAPIC support can actually be determined from the LAPIC version. - physical mode support _removed_, as we can use clustered logical setup to support can support upto a max of 60 CPUs. This is mainly because of the advantage of being able to setup IOAPICs in LowestPrio, when using clustered mode.
The whole stuff is protected by 'Clustered APIC (> 8 CPUs) support (CONFIG_X86_APIC_CLUSTER)' config option under Processor Type and Features. But going forward, we can actually make this as default, as this doesn't affect the systems with less than equal to 8 CPUs (Apart from minor increase in code size and couple of additional checks during boot up), but gives the default support to more than 8 CPU systems.
Please let me know your comments/criticisms about this. I was able to test this successfully on an 8-way with HT(16 logical) CPU systems that I have access to. But, I haven't tested it on x440, or NUMAQ systems. Would love to hear about the effect of this patch on these systems.
Thanks, -Venkatesh
> -----Original Message----- > From: Nakajima, Jun > Sent: Thursday, December 12, 2002 7:06 PM > To: jamesclv@us.ibm.com; Zwane Mwaikambo > Cc: Martin Bligh; John Stultz; Linux Kernel > Subject: RE: [PATCH][2.5][RFC] Using xAPIC apic address space > on !Summit > > > BTW, we are working on a xAPIC patch that supports more than > 8 CPUs in a > generic fashion (don't use hardcode OEM checking). We already > tested it on > two OEM systems with 16 CPUs. > - It uses clustered mode. We don't want to use physical mode > because it does > not support lowest priority delivery mode. > - We also check the version of the local APIC if it's xAPIC > or not. It's > possible that some other x86 architecture (other than P4P) uses xAPIC. > > Stay tuned. > > Jun [unhandled content-type:application/octet-stream]
|  |