Messages in this thread |  | | Date | Tue, 5 Sep 2000 13:06:57 -0700 (PDT) | From | dean gaudet <> | Subject | O_CLOEXEC (was Re: thread rant) |
| |
On Sat, 2 Sep 2000, Alexander Viro wrote:
> On Sat, 2 Sep 2000, Jamie Lokier wrote: > > > dean gaudet wrote: > > > an example of brokenness in the traditional fd API is close-on-exec -- > > > there's a race between open()/socket()/pipe() and fcntl(FD_CLOEXEC) during > > > which if another thread does a fork() it's possible the child will inherit > > > an fd it shouldn't... working around it is painful. the model which > > Really? Like, say it, close() before exec()? > > > > NT/OS2 use for creating a new process scales better in the 99.99% case of > > > stdin/out/err -- you only specify those fds you want to keep in the new > > > process. > > > > An obvious solution presents itself. O_CLOEXEC.
yeah i proposed this and non-blocking inheritance from accept() when i was doing the initial modelling for threaded apache-2.0 about 3 years ago. you need O_CLOEXEC to be inherited through accept() as well. oh, and you also want it on pipe(), socket(), and socketpair() to be complete. (if you search linux-kernel archives you can find that this topic has come up maybe 4 times in 3 years.)
> Even more obvious solution: close what you need to close if you have > sensitive descriptors around. Close-on-exec is a kludge. If you have > sensitive pieces of descriptor table you want to do some other things too > - e.g. make sure that it gets unshared before exec(). Because new > descriptors of that kind are very likely to follow...
if you'll forgive me for using webservers as an example again i can maybe give some motivation.
suppose you're building a threaded server to scale to 20k connections, and you want to support fork()/exec() of CGIs dirctly (rather than through a separate daemon process).
if you don't use close-on-exec, then after fork before exec you have to iterate over the 20k descriptors to close them all -- or you have to walk your own data structures (a list of open sockets) to find what descriptors are open and close them.
if you want to walk your own data structures then these structures need to be coherent at the time of the fork -- which means semaphores to add/delete from the list. this would mean an extra synchronization after accept(). i suspect this would suck worse than iterating over 3..20000 and calling close -- think of all the cache line activity on an SMP box.
if you use O_CLOEXEC then it's not possible for an fd to be open which isn't marked to be closed on the exec, and so you've no extra work to do before exec.
but if you try to use FD_CLOEXEC there's a critical section between the open/socket/accept/pipe/... and the fcntl(). you pretty much have to put it under a semaphore (although you can get away with a rwlock), or iterate over all the descriptors.
opendir() has a similar race, except it's hidden inside libc.
the most portable, and arguably cleaner solution is to use a separate unthreaded process for forking and fd passing... as long as this meets your needs.
fork/exec and threads just don't mix so well with lots of fds. (a spawnve() API with an additional argument to indicate which fds to dup would be better... similar to NT's CreateProcess)
i've a hairbrained idea for replacing fd integers with pointers which might actually scale better if anyone wants to hear it again.
-dean
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org Please read the FAQ at http://www.tux.org/lkml/
|  |