[lkml]   [2010]   [Mar]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [PATCH -tip] introduce sys_membarrier(): process-wide memory barrier (v9)

    * Mathieu Desnoyers <> wrote:

    > I am proposing this patch for the 2.6.34 merge window, as I think it is
    > ready for inclusion.

    It's a bit late for this merge window i think.

    > Here is an implementation of a new system call, sys_membarrier(), which
    > executes a memory barrier on all threads of the current process. It can be
    > used to distribute the cost of user-space memory barriers asymmetrically by
    > transforming pairs of memory barriers into pairs consisting of
    > sys_membarrier() and a compiler barrier. For synchronization primitives that
    > distinguish between read-side and write-side (e.g. userspace RCU, rwlocks),
    > the read-side can be accelerated significantly by moving the bulk of the
    > memory barrier overhead to the write-side.

    Why is this such a low level and still special-purpose facility?

    Synchronization facilities for high-performance threading may want to do a bit
    more than just execute a barrier instruction on another CPU that has a
    relevant thread running.

    You cited signal based numbers:

    > (what we have now, with dynamic sys_membarrier check, expedited scheme)
    > memory barriers in reader: 907693804 reads, 817793 writes
    > sys_membarrier scheme: 4316818891 reads, 503790 writes
    > (dynamic sys_membarrier check, non-expedited scheme)
    > memory barriers in reader: 907693804 reads, 817793 writes
    > sys_membarrier scheme: 8698725501 reads, 313 writes

    Much of that signal handler overhead is i think due to:

    - FPU/SSE context save/restore
    - the need to wake up, run and deschedule all threads

    Instead i'd suggest for you to try to implement user-space RCU speedups not
    via the new sys_membarrier() syscall, but via two new signal extensions:

    - SA_NOFPU: on x86 to skip the FPU/SSE save/restore, for such fast in/out special
    purpose signal handlers? (can whip up a quick patch for you if you want)

    - SA_RUNNING: a way to signal only running threads - as a way for user-space
    based concurrency control mechanisms to deschedule running threads (or, like
    in your case, to implement barrier / garbage collection schemes).

    ( Note: to properly sync back you'll also need an sa_info field to tell
    target tasks how many tasks were woken up. That way a futex can be used
    as a semaphore to signal back to the issuing thread, and make it all
    properly event triggered and nicely scalable. Also, queued signals are a
    must for such a scheme. )

    My estimation is that it will be _much_ faster than the naive signal based
    approach - maybe even quite comparable to an open-coded sys_membarrier():

    - as most of the overhead in a real scenario ought to be the IPI sending and
    latency - not the syscall entry/exit. (with a signal approach we'd still go
    into target thread user-mode, so one more syscall exit+re-entry)

    - or for the common case where there are no other threads running, we are
    just in/out of SA_RUNNING without having to do any synchronization. In that
    case it should be quite close to sys_membarrier() - modulo some minimal
    signal API overhead. [which we could optimize some more, if it's visible in
    your benchmarks.]

    Signals per se are pretty scalable these days - now that most of the fastpaths
    are decoupled from tasklist_lock and everything is RCU-ized.

    Further benefits are:

    - both SA_NOFPU and SA_RUNNING could be used by a _lot_ more user-space
    facilities than just user-space RCU.

    - synergetic effects: growing some real high-performance facility based on
    signals would ensure further signal speedups in the future as well.
    Currently any server app that runs into signal limitations tends to shy
    away from them and use some different (and often inferior) signalling
    scheme. It would be better extend signals with 'lightweight' capabilities
    as well.

    All in one, signals are used by like 99.9% of Linux apps, while
    sys_membarrier() would be used only by [WAG] 0.00001% of them.

    So before we can merge this (at least via the RCU tree, which you have sent it
    to), i'd like to see you try _much_, _MUCH_ harder to fix the very obvious
    signal overhead performance problems you have demoed via the numbers above so

    If _that_ fails, and if we get all the fruits of that, _then_ we might
    perhaps, with a lot of hesitation, concede defeat and think about adding yet
    another syscall.

    I know it's cool to add a brand new syscall - but, unfortunately, in practice
    it doesnt help Linux apps all that much. (at least until we have tools/klibc/
    or so.)

    [ There's also a few small cleanliness details i noticed in your patch: enums
    are a tiny bit nicer for ABIs than #define's, the #ifdef SMP is ugly, etc. -
    but it doesnt really matter much as i think we should concentrate on the
    scalability problems of signals first. ]



     \ /
      Last update: 2010-03-04 17:13    [W:0.031 / U:12.508 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site