Discussion:
stuck in my accept call :-(
sven falempin
2013-12-05 05:22:01 UTC
Permalink
hello:

libevent-2.0.21-stable

https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811

event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);

then i do new in server::serv_sock_accept_pattern and path the arguments to
accept.

But apparently socket has nothing to accept.

It is not easy to produce....

help/insight welcome.
--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\
Yucong Sun
2013-12-05 06:49:47 UTC
Permalink
I would take a blind guess and maybe you need to adjust somaxconn

http://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html

12.14.1.2. kern.ipc.somaxconn

The kern.ipc.somaxconn sysctl(8) variable limits the size of the
listen queue for accepting new TCP connections. The default value of
128 is typically too low for robust handling of new connections on a
heavily loaded web server. For such environments, it is recommended to
increase this value to 1024 or higher. A service such as sendmail(8),
or Apache may itself limit the listen queue size, but will often have
a directive in its configuration file to adjust the queue size. Large
listen queues do a better job of avoiding Denial of Service (DoS)
attacks.
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the arguments to
accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\
***********************************************************************
To unsubscribe, send an e-mail to ***@freehaven.net with
unsubscribe libevent-users in the body.
Tomer Heber
2013-12-05 07:52:52 UTC
Permalink
Hi,

Is your listener socket set to non-blocking? (I guess it's not).

I suggest setting it to non-blocking. Just be prepared to handle error values returned such as: EAGAIN, EWOULDBLOCK etc...

10x,
Tomer.

-----Original Message-----
From: owner-libevent-***@freehaven.net [mailto:owner-libevent-***@freehaven.net] On Behalf Of Yucong Sun
Sent: Thursday, December 05, 2013 8:50 AM
To: libevent-users
Subject: Re: [Libevent-users] stuck in my accept call :-(

I would take a blind guess and maybe you need to adjust somaxconn

http://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html

12.14.1.2. kern.ipc.somaxconn

The kern.ipc.somaxconn sysctl(8) variable limits the size of the listen queue for accepting new TCP connections. The default value of
128 is typically too low for robust handling of new connections on a heavily loaded web server. For such environments, it is recommended to increase this value to 1024 or higher. A service such as sendmail(8), or Apache may itself limit the listen queue size, but will often have a directive in its configuration file to adjust the queue size. Large listen queues do a better job of avoiding Denial of Service (DoS) attacks.
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the
arguments to accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
--
----------------------------------------------------------------------
-----------------------------------------------
() ascii ribbon campaign - against html e-mail /\
***********************************************************************
To unsubscribe, send an e-mail to ***@freehaven.net with
unsubscribe libevent-users in the body.
N�����r��zǧu�ޙ�������ݢj���j����p��
sven falempin
2013-12-05 19:07:54 UTC
Permalink
Post by Tomer Heber
Hi,
Is your listener socket set to non-blocking? (I guess it's not).
I suggest setting it to non-blocking. Just be prepared to handle error
values returned such as: EAGAIN, EWOULDBLOCK etc...
10x,
Tomer.
Nop it was ! and it is still non blocking..

i was calling listen(sock, 8), i will try with higher value.

Still think it is odd.
Post by Tomer Heber
-----Original Message-----
Sent: Thursday, December 05, 2013 8:50 AM
To: libevent-users
Subject: Re: [Libevent-users] stuck in my accept call :-(
I would take a blind guess and maybe you need to adjust somaxconn
http://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html
12.14.1.2. kern.ipc.somaxconn
The kern.ipc.somaxconn sysctl(8) variable limits the size of the listen
queue for accepting new TCP connections. The default value of
128 is typically too low for robust handling of new connections on a
heavily loaded web server. For such environments, it is recommended to
increase this value to 1024 or higher. A service such as sendmail(8), or
Apache may itself limit the listen queue size, but will often have a
directive in its configuration file to adjust the queue size. Large listen
queues do a better job of avoiding Denial of Service (DoS) attacks.
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the
arguments to accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
--
----------------------------------------------------------------------
-----------------------------------------------
() ascii ribbon campaign - against html e-mail /\
***********************************************************************
unsubscribe libevent-users in the body.
--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\
sven falempin
2013-12-05 15:58:09 UTC
Permalink
Post by Yucong Sun
I would take a blind guess and maybe you need to adjust somaxconn
http://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html
12.14.1.2. kern.ipc.somaxconn
The kern.ipc.somaxconn sysctl(8) variable limits the size of the
listen queue for accepting new TCP connections. The default value of
128 is typically too low for robust handling of new connections on a
heavily loaded web server. For such environments, it is recommended to
increase this value to 1024 or higher. A service such as sendmail(8),
or Apache may itself limit the listen queue size, but will often have
a directive in its configuration file to adjust the queue size. Large
listen queues do a better job of avoiding Denial of Service (DoS)
attacks.
ok, i forgot about this one , and i put maxfiles
Post by Yucong Sun
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the arguments
to
Post by sven falempin
accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
--
---------------------------------------------------------------------------------------------------------------------
Post by sven falempin
() ascii ribbon campaign - against html e-mail
/\
***********************************************************************
unsubscribe libevent-users in the body.
--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\
Nick Mathewson
2013-12-06 17:14:09 UTC
Permalink
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the arguments to
accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
I'm not seeing an obvious problem with that code, assuming that
server::serv_sock_accept_pattern has the right signature and is
declared as 'cdecl' if that's necessary. If you're doing anything
unusual with threads, that could account for a socket appearing
readable but having nothing to accept when you look at it .. but you
probably would have thought of that.

A few possibly-not-too-helpful thoughts I have for debugging:

1) use strace or truss or whatever the best FreeBSD syscall logging
tool is to see what exactly is going on with the fd in question. Are
there weird syscalls happening to it? Is something calling accept()
twice somehow?
2) If you're multithreaded, make sure that you've enabled all the
locking stuff that you need in libevent.
3) If you haven't done so already, use helgrind and valgrind to look
for whether some kind of fishy corruption might be going on.
4) Try to come up with a minimal program and/or testing environment
to reproduce the failing case.
5) If it's not too hard, try dropping in evconnlistener in place of
your current listen/accept logic and see if the problem persists. If
not, it's likely an issue in your code somewhere.
6) If you're using any awesome freebsd features like accept filters
that might affect this code path, try turning them off to see whether
that helps.

I hope one of those ideas is helpful. Good luck!

yrs,
--
Nick
***********************************************************************
To unsubscribe, send an e-mail to ***@freehaven.net with
unsubscribe libevent-users in the body.
sven falempin
2013-12-06 18:05:25 UTC
Permalink
Post by Nick Mathewson
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the arguments
to
Post by sven falempin
accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
I'm not seeing an obvious problem with that code, assuming that
server::serv_sock_accept_pattern has the right signature and is
declared as 'cdecl' if that's necessary. If you're doing anything
unusual with threads, that could account for a socket appearing
readable but having nothing to accept when you look at it .. but you
probably would have thought of that.
1) use strace or truss or whatever the best FreeBSD syscall logging
tool is to see what exactly is going on with the fd in question. Are
there weird syscalls happening to it? Is something calling accept()
twice somehow?
it is ktrace :-) it has a binary format you read with kdump. If i reproduce
i gonna check this.
Post by Nick Mathewson
2) If you're multithreaded, make sure that you've enabled all the
locking stuff that you need in libevent.
done :-) was a bit messy but it looks ok now.
Post by Nick Mathewson
3) If you haven't done so already, use helgrind and valgrind to look
for whether some kind of fishy corruption might be going on.
I always valgrind (helgrind ?) .
Post by Nick Mathewson
4) Try to come up with a minimal program and/or testing environment
to reproduce the failing case.
5) If it's not too hard, try dropping in evconnlistener in place of
your current listen/accept logic and see if the problem persists. If
not, it's likely an issue in your code somewhere.
?
Post by Nick Mathewson
6) If you're using any awesome freebsd features like accept filters
that might affect this code path, try turning them off to see whether
that helps.
I hope one of those ideas is helpful. Good luck!
Thanks a lot :-) apparently it was created by a very low backlog (set to 8
in listen call)

So to reproduce and handle the code i will(as soon as i have time) test
with listen (sock, 1) and all those ideas

because rising a limit is OK, but handling the limit is better !
Post by Nick Mathewson
yrs,
--
Nick
***********************************************************************
unsubscribe libevent-users in the body.
--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\
Nick Mathewson
2013-12-06 18:11:24 UTC
Permalink
Post by sven falempin
Post by Nick Mathewson
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the arguments to
accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
I'm not seeing an obvious problem with that code, assuming that
server::serv_sock_accept_pattern has the right signature and is
declared as 'cdecl' if that's necessary. If you're doing anything
unusual with threads, that could account for a socket appearing
readable but having nothing to accept when you look at it .. but you
probably would have thought of that.
1) use strace or truss or whatever the best FreeBSD syscall logging
tool is to see what exactly is going on with the fd in question. Are
there weird syscalls happening to it? Is something calling accept()
twice somehow?
it is ktrace :-) it has a binary format you read with kdump. If i reproduce
i gonna check this.
Post by Nick Mathewson
2) If you're multithreaded, make sure that you've enabled all the
locking stuff that you need in libevent.
done :-) was a bit messy but it looks ok now.
Great. Remember, you need to enable threading before you create any
event_base, and you need to explicitly make any bufferevent or similar
thing that can be used or affected by multiple threads locked.

http://www.wangafu.net/~nickm/libevent-book/Ref1_libsetup.html has
more details than you necessarily want :)
Post by sven falempin
Post by Nick Mathewson
3) If you haven't done so already, use helgrind and valgrind to look
for whether some kind of fishy corruption might be going on.
I always valgrind (helgrind ?) .
Part of the valgrind suite. Like valgrind, but for muliththreading
errors: http://valgrind.org/docs/manual/hg-manual.html
Post by sven falempin
Post by Nick Mathewson
4) Try to come up with a minimal program and/or testing environment
to reproduce the failing case.
5) If it's not too hard, try dropping in evconnlistener in place of
your current listen/accept logic and see if the problem persists. If
not, it's likely an issue in your code somewhere.
?
Libevent provides a wrapper for accepting TCP connections so that you
don't need to do listen/accept yourself:

http://www.wangafu.net/~nickm/libevent-book/Ref8_listener.html


best wishes,
--
Nick
***********************************************************************
To unsubscribe, send an e-mail to ***@freehaven.net with
unsubscribe libevent-users in the body.
sven falempin
2013-12-13 13:49:11 UTC
Permalink
Hello,

I add a bug related to multithread, but it was just some noise.
Just to be even more sure i use evutil_make_socket_nonblocking now.

I am now at like 12hours or more before reproducing the problem, so next
time i will probably focus on lowering the backlog number.

(gdb) bt
#0 0x00000000004acf7c in __sys_accept ()
#1 0x0000000000492ca0 in accept ()
#2 0x00000000004299cb in listener_read_cb (fd=7, what=<optimized out>,
p=<optimized out>) at listener.c:396
#3 0x0000000000427d6e in event_process_active_single_queue
(activeq=0x800867050, base=<optimized out>) at event.c:1350
#4 event_process_active (base=<optimized out>) at event.c:1420
#5 event_base_loop (base=0x800868300, flags=<optimized out>) at
event.c:1621
#6 0x000000000041b0ef in main (argc=10, argv=0x7fffffffdb88) at main.cc:261


My question is:
Maybe adding the error callback would help ?

My fear is, as the core is running at 100% looks like a bug in the kernel
(i mean event a block mode accept does not raise the load like this ?)

As a first workaround i am thinking calling a poll before the accept,
unless someone as a better idea. (Even if i imagine a threaded close call
on fd 7 that should just return Exxx)

thx for insight.
Post by sven falempin
Post by sven falempin
Post by Nick Mathewson
Post by sven falempin
libevent-2.0.21-stable
https://forums.freebsd.org/viewtopic.php?f=35&t=43654&p=242811#p242811
Post by sven falempin
Post by Nick Mathewson
Post by sven falempin
event_assign(&serv_sock_evt, evt_base,
sock, EV_READ | EV_PERSIST,
server::serv_sock_accept_pattern, this);
event_add(&serv_sock_evt, NULL);
then i do new in server::serv_sock_accept_pattern and path the
arguments
Post by sven falempin
Post by Nick Mathewson
Post by sven falempin
to
accept.
But apparently socket has nothing to accept.
It is not easy to produce....
help/insight welcome.
I'm not seeing an obvious problem with that code, assuming that
server::serv_sock_accept_pattern has the right signature and is
declared as 'cdecl' if that's necessary. If you're doing anything
unusual with threads, that could account for a socket appearing
readable but having nothing to accept when you look at it .. but you
probably would have thought of that.
1) use strace or truss or whatever the best FreeBSD syscall logging
tool is to see what exactly is going on with the fd in question. Are
there weird syscalls happening to it? Is something calling accept()
twice somehow?
it is ktrace :-) it has a binary format you read with kdump. If i
reproduce
Post by sven falempin
i gonna check this.
Post by Nick Mathewson
2) If you're multithreaded, make sure that you've enabled all the
locking stuff that you need in libevent.
done :-) was a bit messy but it looks ok now.
Great. Remember, you need to enable threading before you create any
event_base, and you need to explicitly make any bufferevent or similar
thing that can be used or affected by multiple threads locked.
http://www.wangafu.net/~nickm/libevent-book/Ref1_libsetup.html has
more details than you necessarily want :)
Post by sven falempin
Post by Nick Mathewson
3) If you haven't done so already, use helgrind and valgrind to look
for whether some kind of fishy corruption might be going on.
I always valgrind (helgrind ?) .
Part of the valgrind suite. Like valgrind, but for muliththreading
errors: http://valgrind.org/docs/manual/hg-manual.html
Post by sven falempin
Post by Nick Mathewson
4) Try to come up with a minimal program and/or testing environment
to reproduce the failing case.
5) If it's not too hard, try dropping in evconnlistener in place of
your current listen/accept logic and see if the problem persists. If
not, it's likely an issue in your code somewhere.
?
Libevent provides a wrapper for accepting TCP connections so that you
http://www.wangafu.net/~nickm/libevent-book/Ref8_listener.html
best wishes,
--
Nick
***********************************************************************
unsubscribe libevent-users in the body.
--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\
Nick Mathewson
2013-12-13 17:45:50 UTC
Permalink
Post by sven falempin
Hello,
I add a bug related to multithread, but it was just some noise.
Just to be even more sure i use evutil_make_socket_nonblocking now.
I am now at like 12hours or more before reproducing the problem, so next
time i will probably focus on lowering the backlog number.
(gdb) bt
#0 0x00000000004acf7c in __sys_accept ()
#1 0x0000000000492ca0 in accept ()
#2 0x00000000004299cb in listener_read_cb (fd=7, what=<optimized out>,
p=<optimized out>) at listener.c:396
#3 0x0000000000427d6e in event_process_active_single_queue
(activeq=0x800867050, base=<optimized out>) at event.c:1350
#4 event_process_active (base=<optimized out>) at event.c:1420
#5 event_base_loop (base=0x800868300, flags=<optimized out>) at
event.c:1621
#6 0x000000000041b0ef in main (argc=10, argv=0x7fffffffdb88) at main.cc:261
Maybe adding the error callback would help ?
I think that's a pretty good idea.

100% kernel load on accept() is often symptomatic of ENFILE or EMFILE
, and the typical solution -- after raising resource limits as
appropriate, and making sure you don't have a socket leak -- is to do
something in response other than just closing the socket -- like
closing the listener for a while, or until some sockets have closed,
or disabling it, or such.
--
Nick
***********************************************************************
To unsubscribe, send an e-mail to ***@freehaven.net with
unsubscribe libevent-users in the body.
sven falempin
2013-12-19 03:28:40 UTC
Permalink
Post by sven falempin
Post by sven falempin
Hello,
I add a bug related to multithread, but it was just some noise.
Just to be even more sure i use evutil_make_socket_nonblocking now.
I am now at like 12hours or more before reproducing the problem, so next
time i will probably focus on lowering the backlog number.
(gdb) bt
#0 0x00000000004acf7c in __sys_accept ()
#1 0x0000000000492ca0 in accept ()
#2 0x00000000004299cb in listener_read_cb (fd=7, what=<optimized out>,
p=<optimized out>) at listener.c:396
#3 0x0000000000427d6e in event_process_active_single_queue
(activeq=0x800867050, base=<optimized out>) at event.c:1350
#4 event_process_active (base=<optimized out>) at event.c:1420
#5 event_base_loop (base=0x800868300, flags=<optimized out>) at
event.c:1621
#6 0x000000000041b0ef in main (argc=10, argv=0x7fffffffdb88) at
main.cc:261
Post by sven falempin
Maybe adding the error callback would help ?
I think that's a pretty good idea.
100% kernel load on accept() is often symptomatic of ENFILE or EMFILE
, and the typical solution -- after raising resource limits as
appropriate, and making sure you don't have a socket leak -- is to do
something in response other than just closing the socket -- like
closing the listener for a while, or until some sockets have closed,
or disabling it, or such.
--
Nick
***********************************************************************
unsubscribe libevent-users in the body.
I guess it was a thread that was closing the fd meanwhile the accept (after
poll, because i have no errors).
Because it tooks time i did refactoring , and it s only after I refactor
something thats this potential bad close became obvious. At the moment i am
still unsure what is happening :-(

But i wonder about this valgrind errors (i have symbols):

==11532== Conditional jump or move depends on uninitialised value(s)
==11532== at 0x4E1757: stpcpy (in
/home/myself/jsonserver/bin/jsonserver-debug)
==11532== by 0x4C8CE1: tzset_basic (in
/home/myself/jsonserver/bin/jsonserver-debug)
==11532== by 0x1C: ???
==11532== by 0x38353A3130203930: ???
==11532== by 0x3331303220333439: ???
==11532== by 0x42A151: event_changelist_add (evmap.c:656)
==11532== by 0x62072439DF6F8FDB: ???
==11532== by 0x77393F: ??? (in /home/zvn/jsonserver/bin/jsondb-debug)
==11532== by 0x176F: ???
==11532== by 0x7FF00053F: ???
==11532== by 0x7FF000B1F: ???
==11532== by 0x7FF000AB7: ???
==11532== Uninitialised value was created by a stack allocation
==11532== at 0x4C92F0: asctime_r (in
/home/myself/jsonserver/bin/jsonserver-debug)


and another less important here (when calling vsyslog), i blame arglist
void log(int lvl,const char* d,...) {
va_list arglist;
if (d && *d) {
va_start( arglist, d );
if (gconf.debug || gconf.verbose || lvl != LOG_DEBUG) { //drop log
DEBUG if not -d
vsyslog(lvl, d, arglist);
}
va_end(arglist);
}
}


i am talking about this one because it also mention
asctime_r

(i check all my <stack> allocation )

I really have no clue for now.

evmap.c 656 is change = event_changelist_get_or_construct(changelist, fd,
old, fdinfo); in my 2.0.21 stable.
--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\
Loading...