lkml.org 
[lkml]   [2000]   [Aug]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: NTFS-like streams? (Long)
Here is a reply I wrote internally in order to adress alot of the points raised.  I hope it helps shed some light on the debate.

First, I reply Alexander Viro's posts:

> If you want to see some seriously scary code - look at the HFS
>handling of that stuff. And no, I would not bet a dime that it's done
>correctly. The main problem is that rename(), unlink(), etc. are too
>fine-grained for such beasts. It's not that filesutils will barf on
>names. Just suppose that you are saying mv * foo/bar. Shell will
>substitute the whole whorehouse - filename1:attribute1,
>filename1:attribute2, etc. and in the best case mv will call
>rename() on each of them (just how should we interpret "move the
>attribute A of file B into directory foo/bar"?). In the worst it
>will try to _copy_ them and then remove the old ones. Moreover, you
>are going to confuse the hell of VFS and every program out there (let
>alone POSIX, etc.) if rename() on one name will move a bunch of
>other. It just does not happen.

This does not currently happen. If you have a directory filled with
directories, and call mv *, the shell would not expand down. It
removes the abstraction of the "directory-as-container", just as we
would have a "file-as-container" metaphor. getdents would return the
immediate children, not expand down the tree.

> Doing that will mean (modulo internal kernel problems) complete QA
> on every program that has a chance to run on such filesystem. From
> scratch.

Since when has QA been an issue with linux? And why isn't this
feature able to be turned on optionally, and let people report
problems by testing that WANT the feature?

> Nonportable to normal filesystems. Nonportable to any other UNIX. Will
> be painful over NFS.

Normal filesystem? NTFS is more prevelant than ext2 as I remember...
What definition of 'normalcy' are you using? NFS v4 is also adding
support for EA, and looks expandible to named streams, as I remember.
It is at least being looked at. I will have to come back with
references.

> There is a sane and completely portable scheme - use
> directories. It's less brittle and will work with the standard tool
> on normal filesystems. Why not use it?

(1)Because it isn't the abstraction that the user expects, or wants.
The point of named streams is allowing for structured storage that is
slightly different from the file/directory metaphor. Named streams on
a file provide small amounts of metadata, which is TIGHTLY coupled to
the file, rather than the directory acting as an abstract
bucket/container for random files. Hard links are not neccicary in a
named stream namespace, and probably a bad idea. Again, 'normal'
filesystems DO support this metaphor. HFS+ and NTFS. XFS supports EA
which isn't a full generalization. People have found extended
attributes to be a useful feature. The ability to associate metadata
with a file makes for a richer set of possibilities in the computer as
helping people to organize and manipulate information. With the
plethora of information that most people have to deal with, being able
to associate metadata orthogonal to the internal format of the main
stream is a BIG help.

> And what will do cat(1)? tar(1)? cpio(1)? vi(1)? What, kludging the
> thing into each application? What happens when I patch(1) the text
> file with some properties? Are they gone? Preserved? What happens
> when I diff(1) two files? What, BTW, happens with pipes?
> Redirections?

They will ignore it. Pipes have no need for it, they are not a true
filesystem object. Redirections go to the main stream UNLESS a named
stream is specified.

> Linux is UNIX. Extending it may be OK, but ruining the stuff that
> already works (and works _well_) in exchange for, pardon me, No
> Taste, erm, New Technology... No Thanks.

Thoughtless bigotry.

> If NT programmers' code requires things that are incompatible with
> the requirements of UNIX programs - well, you know where to find NT
> if you need it.

Yes, on just about every desktop. Just the place to ignore it.

> Either propose the semantics compatible with the normal UNIX one or
> show that you can and will patch the userland so that it would keep
> working. In _all_ cases. Again, if it boils down to the choice
> between the normal UNIX scripting and k3wlNT3D1T0R - sorry, the
> former wins. Unconditionally.

Perhaps SOME cases will fail. Also, maybe that isn't bad. The
symlink change you made broke applications. Where is the difference?
This is much more useful than a change simply for academic purity.
The real question is how significant is the breakage?

> Good. How about the following implementation of your fd_open():
> mount(name, tmp, "ntfork", 0, NULL);
> sprintf(tmpname, "%s/%s", tmp, field);
> fd = open(tmpname, ......);
> return fd;

> Yep, requires umount(). But notice that this way we * don't need any
> magic syscalls * avoid all questions about out-of-file renaming,
> etc. * don't have

> to do _any_ VFS changes - it's all there. * let the userland decide
> (and explicitly tell us) what it wants. And yes, I mean making that
> case of mount() user-available. Comments? I can implement that for
> HPFS (EAs) in a day or two. As the matter of fact, I would _really_
> like to do that for _HFS_ - that would allow to drop tons of ugly
> code there.

How do you access the main stream after doing that. The file just
magically mutated from a file to a directory. If two applications
have the same file open, suddenly one will lose access to the main
stream, and will have a directory there. This is bad from a
programming/users perspective. Someone later mentions users panicing
at the unknown. If an application can't find their data, and there is
a directory there, what do you think will happen? ;-)


>> Suggestions welcome. What's your interface of choice for a filesystem
>> like HFS that _does_ have resource forks? Whether you like them or not
>> is completely immaterial - they exist.
>>
>> And usability concerns _are_ real concerns. I'm claiming that the best
>> interface for such a filesystem would be
>>
>> open("file", O_RDONLY) - opens the default fork
>> open("file/Icon", O_RDONLY) - opens the Icon fork
>> open("file/Creator"...
>>
>> readdir("file") - lists the resources that the file has
>>
>> and I'm also claiming that the Linux VFS layer actually shouldn't have
>> any fundamental problems with something like this.
>
> Shouldn't or doesn't? I can tell you what the current problems
>_are_.
>
> a) in a _lot_ of places we are required to distinguish between
>directories and non-directories and yes, a lot of things in userland
>depend on that.

I think later someone suggests an extra mode bit. The idea of just
using directories for this also violates this idea. It would require
that every application be rewritten to get the functionality that
named streams offers. Which is much worse than breaking a few apps.

> b) unlink() on such beasts. Welcome to fun. And no, it's not
>rmdir() - here we are removing non-empty object.

Yes, the filesystem has to be able to unlink the children also. It IS
handleable by a FS. Complex locking issues, but it is possible to do.

> c) rename() of normal file to such animal and vice versa.

What do you mean here? You definately couldn't use a hard link and
then delete the old one. link would fail. So you have to copy and
then remove. Which can be forced by having rename return EXDEV.
Slightly shady, but achieves the desired results.

> d) rename() of directory <<--->>

Not allowed. Once John finishes cleaning up the paper, I'll post a
paper that more thoroughly defines these semantics.

> e) propagation of chmod() results

Immediate to the file. Do you propagate chmod to the children of a
directory? And does chmod even work on all filesystems properly?
(NTFS)

> f) _if_ we do unlink() - what should happen with
>delete-upon-the-last-iput() semantics?

This is sticky, but if unlink fails if users have sub-streams open,
then we can do it easily. This is the easiest way in kernel space.
rm could also be modified to do a recursive delete on the attributes
first as a userspace solution.

>That's more or less it. Oh, and have fun with link(2) if some filesystems
>that have forks support it. If you've got a way to handle multiple links
>to object that has children - I'll be glad to hear it.

I personally solve the problem by not allowing named streams to be
hard links. The problem isn't links to objects with mutliple
children, at least on well designed filesystems. It's to circular
links forming, which add a TON of complexity to preventing
inaccessable garbage from filling up the disk.

> Unlike Plan 9 we have a userland that Should Not Be Broken(tm). And we
> have this "POSIX-compliant" thing too... Linus, POSIX has a lot of nasty
> words about mixing files and directories. And I'm afraid that saying "no,
> foo is file, it just happens to have children" won't work - that way you
> are going to screw a lot of userland stuff.

Linux isn't strictly POSIX compliant anymore. Not an issue.

> Again, the _real_ problem is to give a reasonable semantics to unlink(),
> remove-on-last-close, rename() and rmdir() for these beasts. Give me that
> and I'll do VFS support Real Fast(tm). I propose a cop-out - let mount()
> do the job. Then _all_ problems go away, since dubious operations are
> automatically prohibited. If you have something better in mind - go ahead,
> tell what you want.

See (1)

>> I think the above are non-VFS issues to a large degree, and will depend on
>> how the low-level filesystem handles things.
>
> Umm... I don't think so. Check d_invalidate() for some examples. It cares
> for files vs. directories. Has to.

It cares whether a d_entry has children. A reasonable clarification.

>> The _interesting_ thing is what the filesystem does with the "struct
>> inode", for example.
>>
>> I suspect that a filesystem that supports resource forks will just have
>> the same inode for the whole thing. In effect, they would look somewhat
>> like hard links to a normal UNIX application. But the filesystem would
>> save the resource-specific informaiton in the "dentry->d_private" field.
>
>You know what tar(1) will do with you for that, don't you? Same ->st_ino
>with different contents... And unlike procfs, here tar is a Reasonable
>Thing(tm).

I do disagree with having the same inode field for the named streams
as the main file. That could lead to some *massive* ugliness. I
would be interested to see a scheme where that works. I just use
different inodes for the streams. Easy solution, and the concept of
having a node per stream is good. Removing the union, and having a
smaller, lightweight inode strucure would help remove the memory
overhead.

> Oh, yes. Linus, I would _really_ ask you to postpone the activity in that
> direction until
> a) ->revalidate() interface (along with its races) is sorted out
> b) ->getattr() will be in place and used by VFS
> c) icache hashing by ->i_dev issue is sorted out, quite possibly
>along with the ->st_fstype thing

Could you post the discussions on this to the list? I haven't seen
them, and they could be quite enlightening to alot of people.

>> It's great stuff since can tag a file by some kind of property, such as a miniturized
>> icons, etc... and have be be explicitly be associate with the files instead of having
>> that stuff maintained in a database of some sort creating maintenance issues.
>
> Sigh.. WTF does GUI stuff go hand-in-hand with assumption that
>system is single-user one? 1001st time: icons are _NOT_ file properties.
>They depend on user. Get over it, already...

What the fuck(tm) says that this discussion is assumes single user?

Simple refutation of your assumption:

"/usr/share/my_file:kde:zapman@homer.interlan.net:ICON"
"/usr/share/my_file:kde:.default:ICON"
"/usr/share/my_file:gnome:zapman@homer.interlan.net:ICON"

Please refrain from simple minded, hostile refutations... They are
bad for everyone, and do not breed positive discussion.

> UI is not OS. Icon is part of UI, permissions are not.

You are interfacing with the user by denying him permission. It *is*
part of the user interface.

> [the rest of comments snipped to avoid inevitable screams from resident]
> [Mac-lovers; I know where to find *.advocacy cesspool if I'll need it]

See above.


>> The same thing can be conceptually used to create that wet dream of user
>> mounts: going "inside" tar-files by just mounting them as a
>> mini-filesystem on top of the file that is the tar archive. The strongest
>> argument against that is probably the fact that "tar" is not that great a
>> filesystem format ;)
>>
>> However, the "filesystem within a filesystem" approach certainly would
>> require more VFS layer tinkering to get right. It might be a very
>> successful approach, though.
>
>It takes _less_. Linus, the only (and I mean it) issue is that we will
>need to lift the 255-anon-mounts limit. The only problem I have with
>podfuk and friends is their attempt to make mounting automatic ("you've
>looked funny at foo.tar, we mount it"). If application says "I want to
>treat foo.tar as tarfs" (not necessary doing the actual mounting; autofs-like
>scheme will work fine) - no problem. And yes, that's a hell of a powerful
>thing. Potentially there is a file-becoming-directory thing (confused
>userland), but that may be worth saying "fix your userland, then" - amount
>of breakage is probably limited.

You seem to be assuming that one application will be accessing the
file at once. This *is* a mutli-user system we're talking about,
right? If you suddenly change the interpretation between
container/stream for all applications, how is this better, and not
breaking things?

> They will want it for tar(1). Immediately. And for rpm. And for all weird
> shit handled by things like mc(1). Mark my word, you will get a _huge_
> recurring flamewar on hands. With joint letters begging to put file(1)
> into the kernel. Let's leave that to userland. Really. "mark foo as a
> tarfs fodder" - sure. Will do wonders in filemanagers. Putting that
> into the kernel... Not too pleasant.
>
> As for the GNOME looking nice...
> <looking around for GNOME folks>
> <note date=Monday text="don't forget body-armor">
> Just look at the guts of that animal someday. It makes EMACS look
> beautiful and well-designed. I'm serious. No offence to Miguel et.al., but
> GNOME design is just plain Wrong. The underlying model (CORBA) is horribly
> overdesigned and they keep bloating their interfaces. It's a walking
> offence to Occam's Razor, and I mean API, not applications. That's what
> you get when default reaction to "I want foo" is not "hmm... probably can
> be done as combination of bar, baz and quux" but "sure, let's just add a
> new piece to the API". Sigh...

Computer programming is not easy. There is not always an "elegant and
pure solution." GNOME is doing a wonderful job of getting the OS you
dogmatically and rabidly defend accepted in the user community.
Trashing it is NOT a good way of making them friends. Arrogance is
the way to defeat. I have nothing to do with GNOME, but I am angered
by the treatment you gave them there.



On Sat, 12 Aug 2000, Linus Torvalds wrote:

> > need to lift the 255-anon-mounts limit.
>
> [ Thinking about it ]
>
> I don't think you necessarily need to fix even that.

Just a quick check: we are still talking about 2.5, aren't we?

> I'm not positive that you even need a new "struct super_block". You could
> use the same superblock, and just move some more information into the
> vfsmount. Which _is_ easy to allocate, and doesn't have any of the
> anon-mount list issues that the superblock has.

Umm... struct super_block is obese, indeed, but that can be helped.

>> Besides, you _do_ want to have some good way to reach the underlying
>> superblock anyway, even from such a "sub-mount". Just sharing the sb
>> directly would be one easy way.
>>
>> (No, I haven't checked all the details. There may be overriding reasons
>> why you'd not want to re-use the same superblock. And there may be tons of
>> small details too).
>
>Well, for one thing you definitely want different super_operations. Sure,
>we can roll such checks into every method ("is it a real thing or just a
>submount"), but that's just plain ugly. Another thing being, games with
>->s_root will become painful.
>
>Again, I really think that lifting that limit is a good thing for many
>other reasons and if we are talking about 2.5 - I'll do that. If you want
>forks in 2.4.0... professional help may be in order.
>
>As for lifting this limit - about the only place (in kernel) that really
>cares is the icache. That's _very_ easy to deal with - just cache by
>struct super_block * instead of kdev_t. _That_ can be done right now (I
>can submit you such patch tonight if you want it) and IMO it makes sense
>regardless of the limit.
>
>Other place that cares is stat(), but that can be dealt with when we will
>actually start playing with that limit.

This illustrates another point with the idea of mounting files. Is
having another sb any cheaper than having another inode?

>On Sun, 13 Aug 2000, James Sutherland wrote:
>
>> Using colon-separated suffixes seems like a reasonable solution IMO.
>
>No. Sorry, but _that_ is not an option. Dealing with aliases hell on VFAT
>was enough. Yes, NTFS will miss some of the "wonders" (objects are
>different, after all), but even the rest is more than enough.

Aliasing really doesn't seem to be an issue here -- how so?

>On Mon, 14 Aug 2000, James Sutherland wrote:
>
>> > BTW, "same inode" thing is obviously wrong, but who says that we
>> > must go for that?
>>
>> That whole concept had me reaching for the sickbag. I'd just drop the
>> whole "files == directories == mount-points == some sort of magic thing
>> which breaks everything" crap.
>
>FWIW, the only part that makes me sick is "files == directories". _That_
>sucks. "mountpoint" is not a problem. "breaks everything" is mostly due to
>reusing the inumber and I _really_ don't see any reasons to do that.
>Now, having the names that travel along _is_ sick.

Files != directories. They are both containers, but with different semantics.

> They are _not_ distinct in the beginning. OK, NTFS is claimed to support
> links. I bet that they apply to the whole bunch of forks/whatever, not to
> the individual components. rename() will do...?

Yes. They refer to a whole file.

> That's wonderful. So let me do substitution:
>
> What does "you" mean (in context of system files) on multi-user OS, in
> particular on Windows 2000?
>
> BTW, I suspect that a lot of the userland stuff there had been written in
> implicit assumption that it is single-user system.

No, it wasn't. NT is a very multi-user system. 'You' is the current
security context. Just as 'you' under unix is your fs/eu/uid. (And
if you want to get into bigotry, which model has a more complete
version of 'you'?) Win32 may have some problems, but it isn't
completely "single-user assuming" garbage. Yes, some of the
applications are, but who runs word on a time sharing system anyway?
And alot of Linux apps assume single user use also. For end
applications, alot of times it makes sense. Just as an example, look
at StarOffice.

NT is heavily multi-user at the core. It was designed off of VMS
after all... Is it really helpful to bash things you aren't familiar
with off of silly prejudices.

And then on to the rest in a bit...

Michael Zappe

------------------------------------------------
Disclaimer: My opinions almost never overlap
with my employers. Especially that whole liberal
use of profanity thing. You guys can let me out
of the cage now. I just want a little coffee.
Just a little. Please...



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:58    [W:0.188 / U:0.172 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site