Issue in using bufferevent in multithreaded code

[Ashish Bhawsar]

Hi,

I am facing some issues while using bufferevent over OpenSSL in
multithreaded application.

I am using bufferevents to communicate over a SSL connection. In my
application one thread is the
main listener and data reading thread and set of other worker threads.
While main thread is
reading all the data for all the SSL connections, diferent worker pool
threads can write to the underlying
bufferevents associated with main thread's base event and event loop.

I am frequently getting followign error
[event_queue_remove: 0x7fb3ec0037e8(fd 21) not on queue 8]
[evmap.c:372: Assertion nwrite >= 0 failed in evmap_io_del]

I have already enabled the thread-safety features of libevent and openssl.

Can someone please take a look in this issue.

[Nick Mathewson]

On Sat, Feb 15, 2014 at 7:04 AM, Ashish Bhawsar

Post by Ashish Bhawsar
Hi,
I am facing some issues while using bufferevent over OpenSSL in
multithreaded application.
I am using bufferevents to communicate over a SSL connection. In my
application one thread is the
main listener and data reading thread and set of other worker threads. While
main thread is
reading all the data for all the SSL connections, diferent worker pool
threads can write to the underlying
bufferevents associated with main thread's base event and event loop.
I am frequently getting followign error
[event_queue_remove: 0x7fb3ec0037e8(fd 21) not on queue 8]
[evmap.c:372: Assertion nwrite >= 0 failed in evmap_io_del]

This implies that there's some internal error happening inside the
evmap data structure; some inconsistent state concerning the event in
qu3estion.

I'd suspect a locking issue, except for the fact that you say you've
turned locks on.

Post by Ashish Bhawsar
I have already enabled the thread-safety features of libevent and openssl.
Can someone please take a look in this issue.

You could try running it with helgrind or some other thread/locking
debugging tool to see if there's some problem with libevent's locking
here.

You could enable libevent's debug mode (see the reference) before
starting the rest of your program, to see if anything inconsistent is
happening with events in a way that libevent knows how to detect.

You could enable event debug logging, and track what exactly happens
to the event that shows up in the error message; maybe it will be
added more than once with an inconsistent type, or maybe it will get
changed before it's deleted.

But if that doesn't work, your best bet will be to come up with a
minimal test program to exhibit the same behavior, so that others can
examine it too.

(BTW, I apologize in advance, but I'm going to be unusually
unresponsive for the next week or so while I'm traveling.)


hope this helps,