lkml.org 
[lkml]   [2008]   [May]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH] [sched] Fixed CPU hotplug and sched domain handling
Max Krasnyansky wrote:
> First issue is that we're leaking doms_cur. It's allocated in
> arch_init_sched_domains() which is called for every hotplug event.
> So we just keep reallocation doms_cur without freeing it.
> I introduced free_sched_domains() function that cleans things up.
>
> Second issue is that sched domains created by the cpusets are
> completely destroyed by the CPU hotplug events. For all CPU hotplug
> events scheduler attaches all CPUs to the NULL domain and then puts
> them all into the single domain thereby destroying domains created
> by the cpusets (partition_sched_domains).
> The solution is simple, when cpusets are enabled scheduler should not
> create default domain and instead let cpusets do that. Which is
> exactly what the patch does.

Here is more info on this, with debug logs.

Here is initial cpuset setup.
cpus 0-3 balanced, cpus 4-7 non-balanced

cd /dev/cgroup
echo 0 > cpusets.sched_load_balance
mkdir boot
echo 0-3 > boot/cpusets.cpus
echo 1 > boot/cpusets.sched_load_balance
...

-----
CPU0 attaching NULL sched-domain.
CPU1 attaching NULL sched-domain.
CPU2 attaching NULL sched-domain.
CPU3 attaching NULL sched-domain.
CPU4 attaching NULL sched-domain.
CPU5 attaching NULL sched-domain.
CPU6 attaching NULL sched-domain.
CPU7 attaching NULL sched-domain.
CPU0 attaching sched-domain:
domain 0: span 0f
groups: 01 02 04 08
CPU1 attaching sched-domain:
domain 0: span 0f
groups: 02 04 08 01
CPU2 attaching sched-domain:
domain 0: span 0f
groups: 04 08 01 02
CPU3 attaching sched-domain:
domain 0: span 0f
groups: 08 01 02 04
-----

Looks good so far.
Now lets bring cpu7 offline (echo 0 > /sys/devices/system/cpu/cpu7/online)

-----
CPU0 attaching NULL sched-domain.
CPU1 attaching NULL sched-domain.
CPU2 attaching NULL sched-domain.
CPU3 attaching NULL sched-domain.
CPU4 attaching NULL sched-domain.
CPU5 attaching NULL sched-domain.
CPU6 attaching NULL sched-domain.
CPU7 attaching NULL sched-domain.
CPU 7 is now offline
CPU0 attaching sched-domain:
domain 0: span 11
groups: 01 10
domain 1: span 7f
groups: 11 22 44 08
CPU1 attaching sched-domain:
domain 0: span 22
groups: 02 20
domain 1: span 7f
groups: 22 44 08 11
CPU2 attaching sched-domain:
domain 0: span 44
groups: 04 40
domain 1: span 7f
groups: 44 08 11 22
CPU3 attaching sched-domain:
domain 0: span 7f
groups: 08 11 22 44
CPU4 attaching sched-domain:
domain 0: span 11
groups: 10 01
domain 1: span 7f
groups: 11 22 44 08
CPU5 attaching sched-domain:
domain 0: span 22
groups: 20 02
domain 1: span 7f
groups: 22 44 08 11
CPU6 attaching sched-domain:
domain 0: span 44
groups: 40 04
domain 1: span 7f
groups: 44 08 11 22
----

All cpus are now in the single domain.
Same thing happens when cpu7 comes back online.

----
CPU0 attaching NULL sched-domain.
CPU1 attaching NULL sched-domain.
CPU2 attaching NULL sched-domain.
CPU3 attaching NULL sched-domain.
CPU4 attaching NULL sched-domain.
CPU5 attaching NULL sched-domain.
CPU6 attaching NULL sched-domain.
Booting processor 7/8 APIC 0x7
Initializing CPU#7
Calibrating delay using timer specific routine.. 4655.39 BogoMIPS (lpj=9310785)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 3
Intel(R) Xeon(R) CPU E5410 @ 2.33GHz stepping 06
checking TSC synchronization [CPU#3 -> CPU#7]: passed.
CPU0 attaching sched-domain:
domain 0: span 11
groups: 01 10
domain 1: span ff
groups: 11 22 44 88
CPU1 attaching sched-domain:
domain 0: span 22
groups: 02 20
domain 1: span ff
groups: 22 44 88 11
CPU2 attaching sched-domain:
domain 0: span 44
groups: 04 40
domain 1: span ff
groups: 44 88 11 22
CPU3 attaching sched-domain:
domain 0: span 88
groups: 08 80
domain 1: span ff
groups: 88 11 22 44
CPU4 attaching sched-domain:
domain 0: span 11
groups: 10 01
domain 1: span ff
groups: 11 22 44 88
CPU5 attaching sched-domain:
domain 0: span 22
groups: 20 02
domain 1: span ff
groups: 22 44 88 11
CPU6 attaching sched-domain:
domain 0: span 44
groups: 40 04
domain 1: span ff
groups: 44 88 11 22
CPU7 attaching sched-domain:
domain 0: span 88
groups: 80 08
domain 1: span ff
groups: 88 11 22 44
----

As if cpusets do not exist :).
With the patch we now do the right thing when cpus go off/online.

----
CPU0 attaching NULL sched-domain.
CPU1 attaching NULL sched-domain.
CPU2 attaching NULL sched-domain.
CPU3 attaching NULL sched-domain.
CPU4 attaching NULL sched-domain.
CPU5 attaching NULL sched-domain.
CPU6 attaching NULL sched-domain.
CPU7 attaching NULL sched-domain.
CPU0 attaching sched-domain:
domain 0: span 0f
groups: 01 02 04 08
CPU1 attaching sched-domain:
domain 0: span 0f
groups: 02 04 08 01
CPU2 attaching sched-domain:
domain 0: span 0f
groups: 04 08 01 02
CPU3 attaching sched-domain:
domain 0: span 0f
groups: 08 01 02 04

CPU0 attaching NULL sched-domain.
CPU1 attaching NULL sched-domain.
CPU2 attaching NULL sched-domain.
CPU3 attaching NULL sched-domain.
CPU4 attaching NULL sched-domain.
CPU5 attaching NULL sched-domain.
CPU6 attaching NULL sched-domain.
CPU7 attaching NULL sched-domain.
CPU0 attaching sched-domain:
domain 0: span 0f
groups: 01 02 04 08
CPU1 attaching sched-domain:
domain 0: span 0f
groups: 02 04 08 01
CPU2 attaching sched-domain:
domain 0: span 0f
groups: 04 08 01 02
CPU3 attaching sched-domain:
domain 0: span 0f
groups: 08 01 02 04
CPU 7 is now offline

CPU0 attaching NULL sched-domain.
CPU1 attaching NULL sched-domain.
CPU2 attaching NULL sched-domain.
CPU3 attaching NULL sched-domain.
CPU4 attaching NULL sched-domain.
CPU5 attaching NULL sched-domain.
CPU6 attaching NULL sched-domain.
CPU0 attaching sched-domain:
domain 0: span 0f
groups: 01 02 04 08
CPU1 attaching sched-domain:
domain 0: span 0f
groups: 02 04 08 01
CPU2 attaching sched-domain:
domain 0: span 0f
groups: 04 08 01 02
CPU3 attaching sched-domain:
domain 0: span 0f
groups: 08 01 02 04
Booting processor 7/8 APIC 0x7
Initializing CPU#7
Calibrating delay using timer specific routine.. 4655.37 BogoMIPS (lpj=9310749)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 3
Intel(R) Xeon(R) CPU E5410 @ 2.33GHz stepping 06
checking TSC synchronization [CPU#3 -> CPU#7]: passed.

Max


\
 
 \ /
  Last update: 2008-05-28 00:33    [W:1.975 / U:0.896 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site