[lkml]   [2007]   [Feb]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3

    * Ulrich Drepper <> wrote:

    > Ingo Molnar wrote:
    > > in terms of AIO, the best queueing model is i think what the kernel uses
    > > internally: freely ordered, with barrier support.
    > Speaking of AIO, how do you imagine lio_listio is implemented? If
    > there is no asynchronous syscall it would mean creating a threadlet
    > for each request but this means either waiting or creating
    > several/many threads.

    my current thinking is that special-purpose (non-programmable, static)
    APIs like aio_*() and lio_*(), where every last cycle of performance
    matters, should be implemented using syslets - even if it is quite
    tricky to write syslets (which they no doubt are - just compare the size
    of syslet-test.c to threadlet-test.c). So i'd move syslets into the same
    category as raw syscalls: pieces of the raw infrastructure between the
    kernel and glibc, not an exposed API to apps. [and even if we keep them
    in that category they still need quite a bit of API work, to clean up
    the 32/64-bit issues, etc.]

    The size of the async thread pool can be kept in check either from
    user-space (by starting to queue up requests after a certain point of
    saturation without submitting them) or from kernel-space which involves
    waiting (the latter was present in v2 but i temporarily removed it from
    v3). "You have to wait" is the eventual final answer in every
    well-behaved queueing system anyway.

    How things work out with a large number of outstanding threads in real
    apps is still an open question (until someone tries it) but i'm
    cautiously optimistic: in my own (FIO based) measurements syslets beat
    the native KAIO interfaces both in the cached and in the non-cached [==
    many threads] case. I did not expect the latter at all: the non-cached
    syslet codepath is not optimized at all yet, so i expected it to have
    (much) higher CPU overhead than KAIO.

    This means that KAIO is in worse shape than i thought - there's just way
    too much context KAIO has to build up to submit parallel IO contexts.
    Many years of optimizations went into KAIO already, so it's probably at
    its outer edge of performance capabilities. Furthermore, what KAIO has
    to compete against in the syslet case are the synchronous syscalls
    turned async, and more than a decade of optimizations went into all the
    synchronous syscalls. Plus the 'threading overhead of syslets' really
    boils down to 'scheduling overhead' in the end - and we can do over a
    million context-switches a second, per CPU. What killed user-space
    thread-based AIO performance many moons ago wasnt really the threading
    concept itself or scheduling overhead, it was the (then) fragile
    threading implementation of Linux, combined with the resulting
    signal-based AIO code. Catching and handling a single signal is more
    expensive than a context-switch - and signals have legacies attached to
    them that make them hard to scale within the kernel. Plus with syslets
    the 'threading overhead' is optional, it only happens when it has to.

    Plus there's the fundamental killer that KAIO is a /lot/ harder to
    implement (and to maintain) on the kernel side: it has to be implemented
    for every IO discipline, and even for the IO disciplines it supports at
    the moment, it is not truly asynchronous for things like metadata
    blocking or VFS blocking. To handle things like metadata blocking it has
    to resort to non-statemachine techniques like retries - which are bad
    for performance.

    Syslets/threadlets on the other hand, once the core is implemented, have
    near zero ongoing maintainance cost (compared to KAIO pushed into every
    IO subsystem) and cover all IO disciplines and API variants immediately,
    and they are as perfectly asynchronous as it gets.

    So all in one, i used to think that AIO state-machines have a long-term
    place within the kernel, but with syslets i think i've proven myself
    embarrasingly wrong =B-)

    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2007-02-22 08:49    [W:0.030 / U:3.376 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site