[lkml]   [2000]   [Feb]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: accept() improvements for rt signals
Julian Anastasov wrote:
> > > One thing we can do to deal with some of this is to have a new fcntl to
> > > augment F_SETOWN, FASYNC and F_SETSIG. A F_SETFAST() would perform a
> > > setsig and a setown at once, _and_ would accept a poll struct into which
> > > it would fill the current state of the socket. ...
> Sounds good but we have to use this system call after each
> accept() which compared with the fd ownership/flags inheritance
> is slower.

I agree, if the fd created by accept() inherits the ownership & flags
from the listening fd, that gets rid of the race condition when an event
comes in between accept() and F_SETFAST.
But it has a race condition in the case where a single thread listens
and passes connections to a bunch of nonlistening threads;
then you'd need to do a F_SETOWN immediately after accept(),
else an event might go to the wrong place.
If we want that case to be race-free, we might need a system call
that atomically did accept(), F_SETOWN, FASYNC and F_SETSIG together.
(Can anyone channel W. Richard Stevens? He was pretty good at seeing race conditions...)

> > Here's another possible race:
> > User closes fd 5. Events for fd 5 are still in the signal queue.
> > User does accept(), which returns a new fd 5.
> If we use small number of listeners (usually one) we
> can flag that sigwaitinfo() returned info about activity on
> the listener and to accept() the next connections after
> ensuring the rt queue is empty. By this way we know that no new
> fd is reused after our close() and there is no notification
> in the queue for already closed sockets. OK, it is possible
> the accept() to be delayed forever if the rt queue never gets
> empty, i.e. on very busy server. We can't stop the kernel to
> enqueue more and more events.

If you empty the queue with a tight loop, and defer processing
the events, that would let you properly ignore stale fd events
But if there are multiple threads both listening for new connections
and handling old ones, that might give all the pending events to the
thread about to do an accept(). Is that good?

Also, what if there's more than one queue (e.g. a listening
socket with one owner/signum, and connected sockets with a
different owner/signum)? Which queue do you empty? Better
to use an atomic accept()+F_SETFAST combo that generates an
'fd created' signal, then there's no need to empty any queue.

> I think in normal situations we don't need notifications
> from accept(). These events are synchronous. Is there any reason
> for these notifications? If the accepted socket is automatically
> setup for rt signals we don't need notifications from accept().
> But may be they are required if there are still events in the rt
> queue for the closed descriptors and we must know the order of
> the events.

The accept (well, fd creation) events are there so you
don't have to make sure the queue is empty before you call accept().
Not only that - they make it so the user variable that tracks
the poll state of the fd can be fed solely by events, without ever
bypassing the signal queue. This seems like a Good Thing for
eliminating race conditions.

> > Your proposed new ioctl could eliminate both the race you're worried about
> > *and* the one I just pointed out if it also delivered a signal that said
> > "fd 5 created". The user would then discard any events for fd 5
> > received after the close() but before the "fd 5 created" signal.
> But this problem is not only for accept(). We can create
> non-socket descriptors and their events must be notified too.
> This sounds very complex.

e.g. sockets created by open() on special files? I suppose they
would need an atomic open()/F_SETFAST+signal generating combo call,
too. But I think we can leave that for later, since nobody I know is
doing that kind of I/O at the moment.

> We have two variants:
> 1. close() to dequeue the events and to do it very fast.
> 2. If our program closed a fd and sigwaitinfo() returns event
> for listener we have to call accept() after the rt queue is empty
> or at least after the last event currently in the rt queue.

Yes, those are two current workarounds. Zach, which do you use
(or do you use both, or something different)?

> > That "fd 5 created" signal would tell the user code to reset its
> > "fd 5 poll status" variable, which would then be updated by the sigio
> > signals whenever fd 5's poll status changed. No need for an initial
> > call to poll() or a poll struct * in F_SETFAST then.
> In fact, I performed only some tests. But I think we
> even don't need to fallback to poll() on SIGIO.

Perhaps you misunderstand - I wasn't talking about poll() fallback.
I was pointing out that adding the "fd created" signal means
that the signals become a complete specification of the
poll state of the fd; there's no gap between creation and turning
on events where the fd's poll state can change unseen.

> So, what about adding flag to let accept() to copy
> the ownership (f_owner) and fd flags (file->f_flags) from
> the listener to the connected socket. I don't see other races
> if this is the only change to the current kernels.

It's a good idea, but I think I pointed out a race condition
above for server programs that want a single listener socket
plus a bunch of worker sockets with a different owner/signum.
The only solution for that one is an atomic
accept()+F_SETFAST+"fd created" signal queueing call (right?).

> The other
> problem is how to ensure that the events are not for already
> closed descriptors. May be we can play with different signal
> numbers, i.e. we have to check if the signum in the event
> is the expected signum for the descriptor. If not, this is
> event for old fd. We need min 2 signal numbers for this.
> We have to remember which was the last signum used for
> this fd and to set another one using F_SETFAST (if added in
> the kernel). May be there are other better solutions.

I don't like the swapping between two signal numbers idea,
simply because it could be easy to get confused which one was

Thanks for your ideas.

- Dan

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
Please read the FAQ at

 \ /
  Last update: 2005-03-22 13:56    [W:0.096 / U:55.212 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site