lkml.org 
[lkml]   [2004]   [Feb]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: UTF-8 practically vs. theoretically in the VFS API


    On Wed, 18 Feb 2004, Jamie Lokier wrote:
    > Linus Torvalds wrote:
    > > Somebody correctly pointed out that you do not need any out-of-band
    > > encoding mechanism - the very fact that it's an invalid sequence is in
    > > itself a perfectly fine flag. No out-of-band signalling required.
    >
    > Technically this is almost(*) correct,
    >
    > (*) - It's fine until you concatenate two malformed strings. Then the
    > out-of-band signal is lost if the combination is valid UTF-8.

    But that's what you _want_. Having a real out-of-band signal that says
    "this stuff is wrong, because it was wrong at some point in the past", and
    not allowing concatenation of blocks of utf-8 bytes would be _bad_.

    The thing, concatenating two malformed UTF-8 strings is normal behaviour
    in a variety of circumstances, all basically having to do with lower
    levels now knowing about higer-level concepts.

    For example, look at a web-page. Look at how the data comes in: it comes
    as a stream of bytes, with blocking rules that have _nothing_ to do with
    the content (timing, mtu's, extended TCP headers etc etc). That doesn't
    mean that you shouldn't be able to
    - work on the partial results and show them to the user as UTF-8
    - be able to concatenate new stuff as it comes in.

    Having an out-of-band signal for "bad" would literally be a bad idea. If
    you get a valid UTF-8 stream as a result of concatenation, you should
    consider that to be the correct behaviour, or you should CHECK BEFOREHAND
    if you think it is illegal.

    Linus
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2009-11-18 23:46    [W:4.152 / U:0.040 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site