Messages in this thread | | | Date | Tue, 17 Feb 2004 20:35:22 GMT | From | John Bradford <> | Subject | Re: UTF-8 practically vs. theoretically in the VFS API (was: Re: JFS default behavior) |
| |
> > Here on linux-kernel we're saying that if the second program accepts > > any old byte sequence in a filename, it should preserve the byte > > sequence exactly. But any program whose parser-tokeniser is scanning > > UTF-8 is very unlikely to do that - it's just too complicated to say > > some bits of a text stream must be remembered as literal bytes, and > > others must be scanned as multibyte characters. > > So what you are saying is that conversion of invalid multibyte sequences > into non-error wide chars followed by conversion back into UTF-8 can > lead to trouble? *DUH* > > > The holes only arise because software which is interpreting UTF-8 is > > mixed with software which isn't. That's one of the most useful > > features of UTF-8, after all - that's why we use it for filenames. > > The holes only arise because software which is interpreting UTF-8 doesn't > care to do it properly. Software that doesn't interpret it (including the > kernel) doesn't enter the picture at all.
So is your approach to this problem that because the security issue isn't specifically in the kernel, we shouldn't discuss it here? Dispite the fact that there are perfectly good ways of not only working around the issue, but preventing it from even exising, that could be implemented in the kernel?
Filenames _are_ arbitrary strings of bytes, I.E. binary data, and that is how they should be. I totally agree with that, (except that obviously \0 and / have to be treated specially).
However, I don't see why it is any more logical to make the suggestion that filenames generally be treated as UTF-8, IFF they are text at all, than it is to suggest that filename should be arbitrary strings of 32-bit words.
Why not:
* State that filenames are strings of 32-bit words. UCS-4 should be the prefered format for storing text in them, but storing legacy encodings in the low 8 bits is acceptable, (but a Bad Thing for new installations).
* Let legacy applications store 8-bit values in those 32-bit words if they want to, but strongly recommend only 7-bit ASCII values are stored, not values 128-255.
* Create a divide - filesystems that support strings of 32-bit words in their on-disk format, and those that don't. Those that don't can simulate 32-bit words for the new functions that require them, by padding with \0 high bytes.
* Hide all filenames with any values > 255 in them from legacy applications, by not returning data about them in existing legacy kernel functions.
* Introduce new routines to deal with 32-bit filenames, which Unicode applications can use when they need to store non-ASCII, (I.E. non 7-bit).
* Note that UTF-8 stored in the low bytes is still acceptable, (but depreciated in favour of UCS-4).
* Note that this preserves the philosophies of: * No policy in the kernel * Filenames are arbitrary bytestreams, (but now optionally 32-bit ones, not just 8-bit ones!)
* Note that this is very different to my last suggestion, which was fundamentally broken in many ways.
John. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |