Messages in this thread | | | Subject | Re: [regression] cpuset: offlined CPUs removed from affinity masks | From | Valentin Schneider <> | Date | Thu, 16 Jan 2020 18:27:02 +0000 |
| |
On 16/01/2020 17:41, Mathieu Desnoyers wrote: > Hi, > > I noticed the following regression with CONFIG_CPUSET=y. Note that > I am not using cpusets at all (only using the root cpuset I'm given > at boot), it's just configured in. I am currently working on a 5.2.5 > kernel. I am simply combining use of taskset(1) (setting the affinity > mask of a process) and cpu hotplug. The result is that with > CONFIG_CPUSET=y, setting the affinity mask including an offline CPU number > don't keep that CPU in the affinity mask, and it is never put back when the > CPU comes back online. CONFIG_CPUSET=n behaves as expected, and puts back > the CPU into the affinity mask reported to user-space when it comes back > online. > > > * With CONFIG_CPUSET=y (unexpected behavior): > > # echo 0 > /sys/devices/system/cpu/cpu1/online > > % taskset 0x7 ./loop & > % taskset -p $! > pid 1341's current affinity mask: 5 > > # echo 1 > /sys/devices/system/cpu/cpu1/online > > taskset -p $! > pid 1341's current affinity mask: 5 > > kill $! >
As discussed on IRC, this is because we have in sched_setaffinity():
cpuset_cpus_allowed(p, cpus_allowed); cpumask_and(new_mask, in_mask, cpus_allowed);
Another source of issue is that CPUs are taken out of cpusets when hotplugged out, and not put back in when hotplugged back in (except for the root cpuset which follows cpu_active_mask).
Both cpuset.effective_cpus and cpuset.allowed_cpus seem to only span online CPUs:
root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.effective_cpus 0-5 root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.cpus 0-5 root@valsch-juno:~# echo 0 > /sys/devices/system/cpu/cpu3/online [93418.733050] CPU3: shutdown [93418.735815] psci: CPU3 killed (polled 0 ms) root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.cpus 0-2,4-5 root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.effective_cpus 0-2,4-5
The thing is, with CONFIG_CPUSET=n, we can absolutely cope with p->cpus_ptr spanning CPUs that are offline because we still check the active/online mask (is_cpu_allowed()). So one thing I'd like to know is why do cpusets remove offline cpus from their mask? I could see cpuset.allowed containing both online & offline CPUs, and cpuset.effective containing just the online ones.
That way in sched_setaffinity() we can still check for cpuset.allowed, and we still have the online/active check in __set_cpus_allowed_ptr() to deny stupid requests.
| |