lkml.org 
[lkml]   [2023]   [Jan]   [11]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: rseq CPU ID not correct on 6.0 kernels for pinned threads
From
On 2023-01-11 09:52, Mathieu Desnoyers wrote:
> On 2023-01-11 06:26, Florian Weimer wrote:
>> The glibc test suite contains a test that verifies that sched_getcpu
>> returns the expected CPU number for a thread that is pinned (via
>> sched_setaffinity) to a specific CPU.  There are other threads running
>> which attempt to de-schedule the pinned thread from its CPU.  I believe
>> the test is correctly doing what it is expected to do; it is invalid
>> only if one believes that it is okay for the kernel to disregard the
>> affinity mask for scheduling decisions.
>>
>> These days, we use the cpu_id rseq field as the return value of
>> sched_getcpu if the kernel has rseq support (which it has in these
>> cases).
>>
>> This test has started failing sporadically for us, some time around
>> kernel 6.0.  I see failure occasionally on a Fedora builder, it runs:
>>
>> Linux buildvm-x86-26.iad2.fedoraproject.org 6.0.15-300.fc37.x86_64 #1
>> SMP PREEMPT_DYNAMIC Wed Dec 21 18:33:23 UTC 2022 x86_64 x86_64 x86_64
>> GNU/Linux
>>
>> I think I've seen it on the x86-64 builder only, but that might just be
>> an accident.
>>
>> The failing tests log this output:
>>
>> =====FAIL: nptl/tst-thread-affinity-pthread.out=====
>> info: Detected CPU set size (in bits): 64
>> info: Maximum test CPU: 5
>> error: Pinned thread 1 ran on impossible cpu 0
>> error: Pinned thread 0 ran on impossible cpu 0
>> info: Main thread ran on 4 CPU(s) of 6 available CPU(s)
>> info: Other threads ran on 6 CPU(s)
>> =====FAIL: nptl/tst-thread-affinity-pthread2.out=====
>> info: Detected CPU set size (in bits): 64
>> info: Maximum test CPU: 5
>> error: Pinned thread 1 ran on impossible cpu 1
>> error: Pinned thread 2 ran on impossible cpu 0
>> error: Pinned thread 3 ran on impossible cpu 3
>> info: Main thread ran on 5 CPU(s) of 6 available CPU(s)
>> info: Other threads ran on 6 CPU(s)
>>
>> But I also encountered one local failure, but it is rare.  Maybe it's
>> load-related.  There shouldn't be any CPU unplug or anything like that
>> involved here.
>>
>> I am not entirely sure if something is changing CPU affinities from
>> outside the process (which would be quite wrong, but not a kernel bug).
>> But in the past, our glibc test has detected real rseq cpu_id
>> brokenness, so I'm leaning towards that as the cause this time, too.
>
> It can be caused by rseq failing to update the cpu number field on
> return to userspace. Tthis could be validated by printing the regular
> getcpu vdso value and/or the value returned by the getcpu system call
> when the error is triggered, and see whether it matches the rseq cpu id
> value.
>
> It can also be caused by scheduler failure to take the affinity into
> account.
>
> As you also point out, it can also be caused by some other task
> modifying the affinity of your task concurrently. You could print
> the result of sched_getaffinity on error to get a better idea of
> the expected vs actual mask.
>
> Lastly, it could be caused by CPU hotplug which would set all bits
> in the affinity mask as a fallback. As you mention it should not be
> the cause there.
>
> Can you share your kernel configuration ?

Also, can you provide more information about the cpufreq driver and
governor used in your system ? e.g. output of

cpupower frequency-info

and also output of

sysctl kernel.sched_energy_aware

Is this on a physical machine or in a virtual machine ?

Thanks,

Mathieu

>
> Thanks,
>
> Mathieu
>
>>
>> Thanks,
>> Florian
>>
>

--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

\
 
 \ /
  Last update: 2023-03-26 23:36    [W:1.630 / U:0.032 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site