lkml.org 
[lkml]   [2017]   [Jun]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH] workqueue: Ensure that cpumask set for pools created after boot
From
Date


On 06/12/2017 11:14 AM, Tejun Heo wrote:
> Hello,
>
> On Mon, Jun 12, 2017 at 09:47:31AM -0500, Michael Bringmann wrote:
>>> I'm not sure because it doesn't make any logical sense and it's not
>>> right in terms of correctness. The above would be able to enable CPUs
>>> which are explicitly excluded from a workqueue. The only fallback
>>> which makes sense is falling back to the default pwq.
>>
>> What would that look like? Are you sure that would always be valid?
>> In a system that is hot-adding and hot-removing CPUs?
>
> The reason why we're ending up with empty masks is because
> wq_calc_node_cpumask() is assuming that the possible node cpumask is
> always a superset of online (as it should). We can trigger a fat
> warning there if that isn't so and just return false from that
> function.

What would that look like? I should be able to test it on top of the
other changes / corrections.

>>> The only way offlining can lead to this failure is when wq numa
>>> possible cpu mask is a proper subset of the matching online mask. Can
>>> you please print out the numa online cpu and wq_numa_possible_cpumask
>>> masks and verify that online stays within the possible for each node?
>>> If not, the ppc arch init code needs to be updated so that cpu <->
>>> node binding is establish for all possible cpus on boot. Note that
>>> this isn't a requirement coming solely from wq. All node affine (thus
>>> percpu) allocations depend on that.
>>
>> The ppc arch init code already records all nodes used by the CPUs visible in
>> the device-tree at boot time into the possible and online node bindings. The
>> problem here occurs when we hot-add new CPUs to the powerpc system -- they may
>> require nodes that are mentioned by the VPHN hcall, but which were not used
>> at boot time.
>
> We need all the possible (so, for cpus which aren't online yet too)
> CPU -> node mappings to be established on boot. This isn't just a
> requirement from workqueue. We don't have any synchronization
> regarding cpu <-> numa mapping in memory allocation paths either.
>
>> I will run a test that dumps these masks later this week to try to provide
>> the information that you are interested in.
>>
>> Right now we are having a discussion on another thread as to how to properly
>> set the possible node mask at boot given that there is no mechanism to hot-add
>> nodes to the system. The latest idea appears to be adding another property
>> or two to define the maximum number of nodes that should be added to the
>> possible / online node masks to allow for dynamic growth after boot.
>
> I have no idea about the specifics of ppc but at least the code base
> we have currently expect all possible cpus and nodes and their
> mappings to be established on boot.

Hopefully, the new properties will fix the holes in the current implementation
with regard to hot-add.

>
> Thanks.
>

--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com

\
 
 \ /
  Last update: 2017-06-12 20:55    [W:0.092 / U:0.232 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site