lkml.org 
[lkml]   [2020]   [Mar]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: 5.6-rc3: WARNING: CPU: 48 PID: 17435 at kernel/sched/fair.c:380 enqueue_task_fair+0x328/0x440
On Thu, 5 Mar 2020 at 12:29, Christian Borntraeger
<borntraeger@de.ibm.com> wrote:
>
>
> On 05.03.20 10:30, Vincent Guittot wrote:
> > Le mercredi 04 mars 2020 à 20:59:33 (+0100), Christian Borntraeger a écrit :
> >>
> >> On 04.03.20 20:38, Christian Borntraeger wrote:
> >>>
> >>>
> >>> On 04.03.20 20:19, Dietmar Eggemann wrote:
> >>>>> I just realized that this system has something special. Some month ago I created 2 slices
> >>>>> $ head /etc/systemd/system/*.slice
> >>>>> ==> /etc/systemd/system/machine-production.slice <==
> >>>>> [Unit]
> >>>>> Description=VM production
> >>>>> Before=slices.target
> >>>>> Wants=machine.slice
> >>>>> [Slice]
> >>>>> CPUQuota=2000%
> >>>>> CPUWeight=1000
> >>>>>
> >>>>> ==> /etc/systemd/system/machine-test.slice <==
> >>>>> [Unit]
> >>>>> Description=VM production
> >>>>> Before=slices.target
> >>>>> Wants=machine.slice
> >>>>> [Slice]
> >>>>> CPUQuota=300%
> >>>>> CPUWeight=100
> >>>>>
> >>>>>
> >>>>> And the guests are then put into these slices. that also means that this test will never use more than the 2300%.
> >>>>> No matter how much CPUs the system has.
> >>>>
> >>>> If you could run this debug patch on top of your un-patched kernel, it would tell us which task (in the enqueue case)
> >>>> and which taskgroup is causing that.
> >>>>
> >>>> You could then further dump the appropriate taskgroup directory under the cpu cgroup mountpoint
> >>>> (to see e.g. the CFS bandwidth data).
> >>>>
> >>>> I expect more than one hit since assert_list_leaf_cfs_rq() uses SCHED_WARN_ON, hence WARN_ONCE.
> >>>
> >>> That was quick. FWIW, I messed up dumping the cgroup mountpoint (since I restarted my guests after this happened).
> >>> Will retry. See the dmesg attached.
> >>
> >> New occurence (with just one extra debug line)
> >
> > Could you try to add the patch below on top of dietmar's one so we will have the status of
> > each level of the hierarchy ?
> > The 1st level seems ok but something wrong happens while walking the hierarchy
>
> It seems to speed up the issue when I do a compile job in parallel on the host:
>
> Do you also need the sysfs tree?

No that's enough to understand the problem

All child cfs are removed from the list when a cfs is throttled which
means that the first 3 cfs have been removed when
machine.slice/machine-test.slice has been throttled.
But there are added back when we enqueue a task to make sure to go
through the full tree which has probably already happened to

[ 87.428277] CPU1
path=/machine.slice/machine-test.slice/machine-qemu\x2d14\x2dtest11.
on_list=1 nr_running=1 throttled=0 p=[CPU 2/KVM 2621]
The group entity has been removed from the leaf list when parent has
been throttled but will not be added because nr_running > 1 [
87.428285] CPU1
path=/machine.slice/machine-test.slice/machine-qemu\x2d14\x2dtest11.
on_list=0 nr_running=3 throttled=0 p=[CPU 2/KVM 2621]
This one has been removed when throttled but already added back
because of a previous enqueue_task [ 87.428288] CPU1
path=/machine.slice/machine-test.slice on_list=1 nr_running=1
throttled=1 p=[CPU 2/KVM 2621]
This one has also been added during a previous enqueue_task on the
throttled child above [ 87.428291] CPU1 path=/machine.slice
on_list=1 nr_running=0 throttled=0 p=[CPU 2/KVM 2621]
[ 87.428301] CPU1 path=/ on_list=1 nr_running=1 throttled=0 p=[CPU 2/KVM 2621]

After we added the 1st cgroup, we don't add other cfs to finish the
full hierarchy

\
 
 \ /
  Last update: 2020-03-05 13:15    [W:0.087 / U:8.156 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site