Messages in this thread | | | Subject | Re: [PATCH] workqueue: Ensure that cpumask set for pools created after boot | From | Michael Bringmann <> | Date | Mon, 12 Jun 2017 12:10:49 -0500 |
| |
On 06/12/2017 11:14 AM, Tejun Heo wrote: > Hello, > > On Mon, Jun 12, 2017 at 09:47:31AM -0500, Michael Bringmann wrote: >>> I'm not sure because it doesn't make any logical sense and it's not >>> right in terms of correctness. The above would be able to enable CPUs >>> which are explicitly excluded from a workqueue. The only fallback >>> which makes sense is falling back to the default pwq. >> >> What would that look like? Are you sure that would always be valid? >> In a system that is hot-adding and hot-removing CPUs? > > The reason why we're ending up with empty masks is because > wq_calc_node_cpumask() is assuming that the possible node cpumask is > always a superset of online (as it should). We can trigger a fat > warning there if that isn't so and just return false from that > function.
What would that look like? I should be able to test it on top of the other changes / corrections.
>>> The only way offlining can lead to this failure is when wq numa >>> possible cpu mask is a proper subset of the matching online mask. Can >>> you please print out the numa online cpu and wq_numa_possible_cpumask >>> masks and verify that online stays within the possible for each node? >>> If not, the ppc arch init code needs to be updated so that cpu <-> >>> node binding is establish for all possible cpus on boot. Note that >>> this isn't a requirement coming solely from wq. All node affine (thus >>> percpu) allocations depend on that. >> >> The ppc arch init code already records all nodes used by the CPUs visible in >> the device-tree at boot time into the possible and online node bindings. The >> problem here occurs when we hot-add new CPUs to the powerpc system -- they may >> require nodes that are mentioned by the VPHN hcall, but which were not used >> at boot time. > > We need all the possible (so, for cpus which aren't online yet too) > CPU -> node mappings to be established on boot. This isn't just a > requirement from workqueue. We don't have any synchronization > regarding cpu <-> numa mapping in memory allocation paths either. > >> I will run a test that dumps these masks later this week to try to provide >> the information that you are interested in. >> >> Right now we are having a discussion on another thread as to how to properly >> set the possible node mask at boot given that there is no mechanism to hot-add >> nodes to the system. The latest idea appears to be adding another property >> or two to define the maximum number of nodes that should be added to the >> possible / online node masks to allow for dynamic growth after boot. > > I have no idea about the specifics of ppc but at least the code base > we have currently expect all possible cpus and nodes and their > mappings to be established on boot.
Hopefully, the new properties will fix the holes in the current implementation with regard to hot-add.
> > Thanks. >
-- Michael W. Bringmann Linux Technology Center IBM Corporation Tie-Line 363-5196 External: (512) 286-5196 Cell: (512) 466-0650 mwb@linux.vnet.ibm.com
| |