Messages in this thread | ![/](/images/icornerl.gif) | | From | Florian Weimer <> | Subject | Re: rseq + membarrier programming model | Date | Mon, 13 Dec 2021 21:12:12 +0100 |
| |
* Mathieu Desnoyers:
> ----- On Dec 13, 2021, at 2:29 PM, Florian Weimer fweimer@redhat.com wrote: > >> * Mathieu Desnoyers: >> >>>> Could it fall back to >>>> MEMBARRIER_CMD_GLOBAL instead? >>> >>> No. CMD_GLOBAL does not issue the required rseq fence used by the >>> algorithm discussed. Also, CMD_GLOBAL has quite a few other shortcomings: >>> it takes a while to execute, and is incompatible with nohz_full kernels. >> >> What about using sched_setcpu to move the current thread to the same CPU >> (and move it back afterwards)? Surely that implies the required sort of >> rseq barrier that MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ with >> MEMBARRIER_CMD_FLAG_CPU performs? > > I guess you refer to using sched_setaffinity(2) there ? There are various > reasons why this may fail. For one, the affinity mask is a shared global > resource which can be changed by external applications.
So is process memory …
> Also, setting the affinity is really just a hint. In the presence of > cpu hotplug and or cgroup cpuset, it is known to lead to situations > where the kernel just gives up and provides an affinity mask including > all CPUs.
How does CPU hotplug impact this negatively?
The cgroup cpuset issue clearly is a bug.
> Therefore, using sched_setaffinity() and expecting to be pinned to > a specific CPU for correctness purposes seems brittle.
I'm pretty sure it used to work reliably for some forms of concurrency control.
> But _if_ we'd have something like a sched_setaffinity which we can > trust, yes, temporarily migrating to the target CPU, and observing that > we indeed run there, would AFAIU provide the same guarantee as the rseq > fence provided by membarrier. It would have a higher overhead than > membarrier as well.
Presumably a signal could do it as well.
>> That is possible even without membarrier, so I wonder why registration >> of intent is needed for MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ. > > I would answer that it is not possible to do this _reliably_ today > without membarrier (see above discussion of cpu hotplug, cgroups, and > modification of cpu affinity by external processes). > > AFAIR, registration of intent for MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ > is mainly there to provide a programming model similar to private expedited > plain and core-sync cmds. > > The registration of intent allows the kernel to further tweak what is > done internally and make tradeoffs which only impact applications > performing the registration.
But if there is no strong performance argument to do so, this introduces additional complexity into userspace. Surely we could say we just do MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ at process start and document failure (in case of seccomp etc.), but then why do this at all?
>>> In order to make sure the programming model is the same for expedited >>> private/global plain/sync-core/rseq membarrier commands, we require that >>> each process perform a registration beforehand. >> >> Hmm. At least it's not possible to unregister again. >> >> But I think it would be really useful to have some of these barriers >> available without registration, possibly in a more expensive form. > > What would be wrong with doing a membarrier private-expedited-rseq > registration on libc startup, and exposing a glibc tunable to allow > disabling this ?
The configurations that need to be supported go from “no rseq“/“rseq” to “no rseq“/“rseq”/“rseq with membarrier”. Everyone now needs to think about implementing support for all three instead just the obvious two.
Thanks, Florian
| ![\](/images/icornerr.gif) |