libevent2 and delete-after-close with kqueue

[Adrian Chadd]

Hi all,

I'm trying to chase down a bug with libevent2-trunk.

The TL;DR is that I have ktrace traces from FreeBSD showing that an FD
delete event is being kqueued to a kqueue FD after the FD has been
closed in that thread. It's responsible for ENOTCAPABLE showing up.
Something like EBADF just drops through; ENOTCAPABLE currently causes
the IO loop to terminate. This is with a multi-threaded server, but
there's one event base per thread and one accept FD per event base.

It _looks_ like the evhttp / bufferevent code is doing the right thing
- ie, it's deleting the events (which should delete them from the
kqueue list) before calling close(). But there's a bunch of places
that I have to actually go and check. Unfortunately "EV_DELETE"
entries are being left on the kqueue list and not removed. The only
way to know that these need to be removed is to actually call the
event deletion function from evutil.c when the socket is closed.
Teaching evutil_closesocket() would do it, but it would require an
eventbase. It also implies that a socket is only in one event base -
it's quite possible servers have an FD in multiple event bases. Ugh.

What do people think?


-a
***********************************************************************
To unsubscribe, send an e-mail to ***@freehaven.net with
unsubscribe libevent-users in the body.

[Nick Mathewson]

Post by Adrian Chadd
Hi all,
I'm trying to chase down a bug with libevent2-trunk.
The TL;DR is that I have ktrace traces from FreeBSD showing that an FD
delete event is being kqueued to a kqueue FD after the FD has been
closed in that thread. It's responsible for ENOTCAPABLE showing up.
Something like EBADF just drops through; ENOTCAPABLE currently causes
the IO loop to terminate. This is with a multi-threaded server, but
there's one event base per thread and one accept FD per event base.
It _looks_ like the evhttp / bufferevent code is doing the right thing
- ie, it's deleting the events (which should delete them from the
kqueue list) before calling close(). But there's a bunch of places
that I have to actually go and check. Unfortunately "EV_DELETE"
entries are being left on the kqueue list and not removed. The only
way to know that these need to be removed is to actually call the
event deletion function from evutil.c when the socket is closed.
Teaching evutil_closesocket() would do it, but it would require an
eventbase. It also implies that a socket is only in one event base -
it's quite possible servers have an FD in multiple event bases. Ugh.
What do people think?

Ow, that's annoying.

One thing to consider is that there's a parallel issue with closing
fds with an epoll-based backend. (Look at the documentation about the
difference between the epoll and the epoll-with-changelist backends.)
Possibilities that I can see include:

* Have a kqueue-without-changelist backend that flushes all
EV_DELETEs immediately.

* Have an API like event_finalize that removes any pending
EV_DELETEs as needed.

* Have a "delete event and close fd" API.

Obviously, these possibilities are maybe not so good. Let's see if we
can think of others.



cheers,

[Adrian Chadd]

Post by Nick Mathewson

Post by Adrian Chadd
Hi all,
I'm trying to chase down a bug with libevent2-trunk.
The TL;DR is that I have ktrace traces from FreeBSD showing that an FD
delete event is being kqueued to a kqueue FD after the FD has been
closed in that thread. It's responsible for ENOTCAPABLE showing up.
Something like EBADF just drops through; ENOTCAPABLE currently causes
the IO loop to terminate. This is with a multi-threaded server, but
there's one event base per thread and one accept FD per event base.
It _looks_ like the evhttp / bufferevent code is doing the right thing
- ie, it's deleting the events (which should delete them from the
kqueue list) before calling close(). But there's a bunch of places
that I have to actually go and check. Unfortunately "EV_DELETE"
entries are being left on the kqueue list and not removed. The only
way to know that these need to be removed is to actually call the
event deletion function from evutil.c when the socket is closed.
Teaching evutil_closesocket() would do it, but it would require an
eventbase. It also implies that a socket is only in one event base -
it's quite possible servers have an FD in multiple event bases. Ugh.
What do people think?

Ow, that's annoying.
One thing to consider is that there's a parallel issue with closing
fds with an epoll-based backend. (Look at the documentation about the
difference between the epoll and the epoll-with-changelist backends.)
* Have a kqueue-without-changelist backend that flushes all
EV_DELETEs immediately.
* Have an API like event_finalize that removes any pending
EV_DELETEs as needed.
* Have a "delete event and close fd" API.
Obviously, these possibilities are maybe not so good. Let's see if we
can think of others.

I think the "clean" thing to do is to have a "delete events for this
FD from the changelist in this event base" API.

If people run one FD per event base then it'll translate easily.

If people run a handful of shared FDs across all event bases (eg
accept sockets) then they'll have to add some serialisation to handle
that case.

We'd then have to liberally sprinkle it around the codebase - ie,
wherever evutil_closesocket() is called - but are those all guaranteed
to be used as "one FD per event base" ?


-a
***********************************************************************
To unsubscribe, send an e-mail to ***@freehaven.net with
unsubscribe libevent-users in the body.