lkml.org 
[lkml]   [2008]   [Feb]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: Integration of SCST in the mainstream Linux kernel
    From
    Date
    On Mon, 2008-02-04 at 11:06 -0800, Nicholas A. Bellinger wrote:
    > On Mon, 2008-02-04 at 10:29 -0800, Linus Torvalds wrote:
    > >
    > > On Mon, 4 Feb 2008, James Bottomley wrote:
    > > >
    > > > The way a user space solution should work is to schedule mmapped I/O
    > > > from the backing store and then send this mmapped region off for target
    > > > I/O.
    > >
    > > mmap'ing may avoid the copy, but the overhead of a mmap operation is
    > > quite often much *bigger* than the overhead of a copy operation.
    > >
    > > Please do not advocate the use of mmap() as a way to avoid memory copies.
    > > It's not realistic. Even if you can do it with a single "mmap()" system
    > > call (which is not at all a given, considering that block devices can
    > > easily be much larger than the available virtual memory space), the fact
    > > is that page table games along with the fault (and even just TLB miss)
    > > overhead is easily more than the cost of copying a page in a nice
    > > streaming manner.
    > >
    > > Yes, memory is "slow", but dammit, so is mmap().
    > >
    > > > You also have to pull tricks with the mmap region in the case of writes
    > > > to prevent useless data being read in from the backing store. However,
    > > > none of this involves data copies.
    > >
    > > "data copies" is irrelevant. The only thing that matters is performance.
    > > And if avoiding data copies is more costly (or even of a similar cost)
    > > than the copies themselves would have been, there is absolutely no upside,
    > > and only downsides due to extra complexity.
    > >
    >
    > The iSER spec (RFC-5046) quotes the following in the TCP case for direct
    > data placement:
    >
    > " Out-of-order TCP segments in the Traditional iSCSI model have to be
    > stored and reassembled before the iSCSI protocol layer within an end
    > node can place the data in the iSCSI buffers. This reassembly is
    > required because not every TCP segment is likely to contain an iSCSI
    > header to enable its placement, and TCP itself does not have a
    > built-in mechanism for signaling Upper Level Protocol (ULP) message
    > boundaries to aid placement of out-of-order segments. This TCP
    > reassembly at high network speeds is quite counter-productive for the
    > following reasons: wasted memory bandwidth in data copying, the need
    > for reassembly memory, wasted CPU cycles in data copying, and the
    > general store-and-forward latency from an application perspective."
    >
    > While this does not have anything to do directly with the kernel vs. user discussion
    > for target mode storage engine, the scaling and latency case is easy enough
    > to make if we are talking about scaling TCP for 10 Gb/sec storage fabrics.
    >
    > > If you want good performance for a service like this, you really generally
    > > *do* need to in kernel space. You can play games in user space, but you're
    > > fooling yourself if you think you can do as well as doing it in the
    > > kernel. And you're *definitely* fooling yourself if you think mmap()
    > > solves performance issues. "Zero-copy" does not equate to "fast". Memory
    > > speeds may be slower that core CPU speeds, but not infinitely so!
    > >
    >
    > >From looking at this problem from a kernel space perspective for a
    > number of years, I would be inclined to believe this is true for
    > software and hardware data-path cases. The benefits of moving various
    > control statemachines for something like say traditional iSCSI to
    > userspace has always been debateable. The most obvious ones are things
    > like authentication, espically if something more complex than CHAP are
    > the obvious case for userspace. However, I have thought recovery for
    > failures caused from communication path (iSCSI connections) or entire
    > nexuses (iSCSI sessions) failures was very problematic to expect to have
    > to potentially push down IOs state to userspace.
    >
    > Keeping statemachines for protocol and/or fabric specific statemachines
    > (CSM-E and CSM-I from connection recovery in iSCSI and iSER are the
    > obvious ones) are the best canidates for residing in kernel space.
    >
    > > (That said: there *are* alternatives to mmap, like "splice()", that really
    > > do potentially solve some issues without the page table and TLB overheads.
    > > But while splice() avoids the costs of paging, I strongly suspect it would
    > > still have easily measurable latency issues. Switching between user and
    > > kernel space multiple times is definitely not going to be free, although
    > > it's probably not a huge issue if you have big enough requests).
    > >
    >

    Then again, having some data-path for software and hardware bulk IO
    operation of storage fabric protocol / statemachine in userspace would
    be really interesting for something like an SPU enabled engine for the
    Cell Broadband Architecture.

    --nab





    \
     
     \ /
      Last update: 2008-02-04 20:23    [W:0.028 / U:91.872 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site