lkml.org 
[lkml]   [1997]   [Aug]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: UTF-8, OSTA-UDF [why?], Unicode, and miscellaneous gibberish
On Tue, 19 Aug 1997, Teunis Peters wrote:

> > ...and people use native charsets/encodings for a long time already --
> > and then Unicode appeared to "make it possible". Traditionally everything
> > network-related was supposed to either be ASCII-only or use MIME charsets
> > definitions. It worked fine in Russia and Japan (I have no information
> > about China or Korea), but now Unicode supporters are trying to push
> > "mandatory" Unicode into HTML. They completely ignore that HTTP is never
>
> FWIW - 3 standard encodings in Japanese.... And no way to tell the
> difference. For me anyways Unicode-2.0 [ISO-whatever actually] makes life
> MUCH easier.... I can't really afford to try and hunt down all the
> miriads of encodings anyways.

It's easier for people who are writing GUI software for languages other
than iso8859-1-supporting ones while their native languages are in
iso8859-1 set. For everyone else it's yet another encoding to support, and
the worst one of existing for their language.

>
> AFAIK Chinese has about 4-5 encodings, not counting countries that
> incorporate other character sets as well (Korean, Japanese <somewhat>, and
> so on)
>
> I want Unicode in _ALL_ string actions in my GUI.

What about non-GUI programs? Now all data is the same, and there is no
conversion necesary. Unicode introduces conversion on user interface
unless it's used at every kind of processing internally, thus preserving
sane processing modesl but making unusable absolutely every piece of
software ever written with assumption that any text can be processed byte
by byte regardless of the "real" size of characters.

> Not just WWW.
> Besides, I don't use WWW all that often (and character maps CAN be
> afforded there - they are too slow in GUI)
>

[skipped]

> > > Not that it matters but I think as long as filenames from 16bit+
> > > filesystems should be encoded into UTF-8 before being passed to the user.
> >
> > ...thus requiring to distinguish them from "normal" data everywhere and
> > breaking every piece of software that should treat data in files as
> > filenames (say, "make").
>
> No - this should work fine.

really? If I'll use filenames how they are represented in filesystem (say,
UTF-8) and the rest of text in 16-bit encoding, it will be impossible to
edit unless the editor knows where filenames are (I'm sure, it can be
written in Visual Basic and will be called a new advanced development
environment), or everything should be in 16-bit (including zeroes -- goodbye,
ed/sed/awk).

> It sure beats English-oriented OS's as well

Actually I can use _any_ language there already -- and I want to see a
person who has a need for more than one non-English language in Makefile.

> :)
>
> It's just an 8-bit filename...

8-bit? not 16? and not 32? So, I can't process it as a normal text and
should convert every time?

> it's not like Makefile/makefile gets
> scrambled or anything. [or .c/.C/.c++/.java/... the rest of the filename
> can be 8bit for what it matters].
>
> In other words : EXPLAIN!
>
> I was just trying to figure out how 16-bit filesystems are translated in
> Linux.... Why not UTF-8 encode them?

Because everyone will have to encode/decode them on every file operation
if they really want to preserve system calls. Should I explain why it's
important to have trailing 8-bit zero in all strings that are passed to
kernel? But UTF-8 is unusable as the internal format -- even regexps on it
will become a monster. So, again hello horrible 8 -> 16 -> 8
conversions on every operation with text...

[skipped]

> > that fond of Unicode anyway).
>
> Heck - It _STILL_ beats english-only.

There is no such thing as anything English-only. It's in the past, and
everything that can't pass bytes transparently should be fixed if not
everything is fixed already. No one needed Unicode for that though.

> And for that matter all the
> hundreds of character mappings.... which are expensive docs.

Why do you need them? Just keep things transparent. Don't add GUI or
natural-language processing to every program, and you will never have to
see them -- it's a bad idea to slap GUI on everything.

> One could do worse than Unicode-2.0.

...and one can do better. If any reasonable multi-encoding extendable
multi-language document representatin system will be developed, I'll
switch to it, and Unicode will only cause problems -- I believe that
any reasonable system should be able to just take a set of encodings and
make some wrapping for them that will allow them to coexist in one
document (yes, I prefer to have charset state in a document rather than
"stateless" encoding with the "alphabet" of this size). _That_ will beat
Unicode.

> IBM-PC character set anyone?

You think, that is the only alternative?

> And hey, the trick is in the translation anyways. Unicode [the 32bit
> variant Unicode-2.x] is a handy way of storing international strings. Who
> says the keyboard has to be US-english or the display ISO-8859-1?
>
> The only written character set I've really seen any problems with is
> Chinese (the 'common' written form).
> [apparently Russian has problems but
> I don't know why? I thought they HAD a fixed alphabet!]

And _that_ is the problem -- Russian alphabet is too simple to require
such a huge system to support it. I am not going to switch to 16 or 32
bits to use Russian that has such a huge number as 33 letters on 2
registers (total 66) and lose compatibility with everything but NTFS. If
anyone is going to show that they like Unicode so much, abandon your 8-bit
ASCII and start using 16-bit one (with zero as high byte, so it will be
the Unicode in its purest form), and I'll see how long will you be able to
keep using that. Please, no cheating as "our beloved Americans can omit
high zero bytes, foreigners use UTF-8 mapping".

> Sorry to mess lkernel up with this but I have a feeling this whole debate
> needs to be figured out... I was just curious how translations from
> 16bit+ filesystems was handled?
>
> Don't say [cut top 8 bits] or [just leave it].... Both solutions mess
> things up. I vote UTF-8 translation.

...and a need to make sense from it in every program? IMHO that will
translate into "ASCII will remain readable, iso8859-1 will be
kinda readable, everything else will screw up unless everyone will switch
to UTF-8 as the internal encoding". Note that even Windows NT doesn't use
UTF-8 internally -- it either keeps things 16-bit or breaks its own
standard and cheerfully strips high byte. That laso means that 16-bit
filesystems are quite unusable there, too, and can be treated as
vaporware.

If someone really wants to handle such things, they should make a loadable
translation interface that will be loaded at the time of mounting, just
like how loop mounting is handled now.

> This is _JUST_ for 16bit+ filesystems. Not 8bit filesystems like ext2 or
> the like.
>
> If there is going to be no standard at least one should be able to probe
> the filesystem type and apply an appropriate translation (eg: this FS is
> vfat ergo std. 16bit Win95-encoded Unicode translated to UTF-8)
>
> Incidentally, the first 7 bits are still normal in UTF-8 so all those
> english filenames (Makefile for example) are NOT touched!!!!!
>
> Does that explain what I said earlier?
>
> Have a nice day, eh?
> - Teunis
>
>
--
Alex


\
 
 \ /
  Last update: 2005-03-22 13:40    [from the cache]
©2003-2011 Jasper Spaans