From toby.douglass at summerblue.net Sat Mar 3 16:18:59 2007 From: toby.douglass at summerblue.net (Toby Douglass) Date: Sat Mar 3 16:17:34 2007 Subject: [Libevent-users] IOCP Message-ID: <45E9E643.7040505@summerblue.net> Okay, testing now complete. I've written a test/demo app which is a client-server arrangement, where the server sits there, the client then opens up as many connects as it can, and then both sides proceed to endlessly issue reads and writes on each socket in turn. This all works absolutely great, except... ...I'm only getting just a little over 2,000 sockets, here at home. I'll release tomorrow, you guys can try it out on your machines and see what you get. From provos at citi.umich.edu Sat Mar 3 23:18:26 2007 From: provos at citi.umich.edu (Niels Provos) Date: Sat Mar 3 23:18:43 2007 Subject: [Libevent-users] source repo queries In-Reply-To: <87ps7uy8ms.wl%rnyberg@murmeldjur.se> References: <87ps7uy8ms.wl%rnyberg@murmeldjur.se> Message-ID: <850f7cbe0703032018j32b833c7k1667c969a4b01c4@mail.gmail.com> I forgot to tag release 1.3; I tagged 1.3a and also 1.3b I just released. You could just download the different versions from http://www.monkey.org/~provos/libevent/ to see what made it into each version. trunk is new stuff that might not be stable enough for a release yet; patches are branches for each release that allow me to manually integrate changes or small fixes. Niels. On 2/28/07, Richard Nyberg wrote: > Hello Niels. > > I've included libevent in a project and want to follow libevent > development closely. > > Could you explain how you use the patches-* branches and trunk? > > You have tags for some previous releases, but not for 1.3 or 1.3a. > Is there another way to see exactly what code went in a particular > release? > > Regards, > -Richard > _______________________________________________ > Libevent-users mailing list > Libevent-users@monkeymail.org > http://monkey.org/mailman/listinfo/libevent-users > > From toby.douglass at summerblue.net Sun Mar 4 10:50:16 2007 From: toby.douglass at summerblue.net (Toby Douglass) Date: Sun Mar 4 10:48:52 2007 Subject: [Libevent-users] libiocp 1.10 release Message-ID: <45EAEAB8.9000705@summerblue.net> http://www.summerblue.net/computing/libiocp/index.html From steveu at coppice.org Mon Mar 5 09:16:14 2007 From: steveu at coppice.org (Steve Underwood) Date: Mon Mar 5 09:16:22 2007 Subject: [Libevent-users] Incremental timers for streaming media Message-ID: <45EC262E.8010405@coppice.org> Hi all, Currently libevent only supports setting timers relative to the time at which they are set. Used directly, this is unsuitable for streaming media, which require packets sent at a well controlled overall rate. I have been using code that reads the clock in the timer event service routine, and makes any necessary adjustments to the next timeout request. This works fine. However, it seems like this functionality ought to be in libevent itself, as streaming is a common requirement. I have added a new function to my local copy of libevent - event_add_incremental() - which sets a timeout relative to the previous deadline, rather than relative to the current (typically slightly later) time. I use event_add() for the first timeout, to set an initial baseline time for the increments. What is incomplete right now is provision for overload situations, where the machine can't keep up with the event servicing, and things fall further and further behind. Falling slightly behind, and then catching up, is quite normal. That means simply rejecting requests for a deadline which is in the past is not satisfactory. Currently I am applying some latitude, and rejecting requests for deadlines which are seriously in the past. I works for my needs, but it doesn't seem a very "complete" solution. I suspect others have faced similar issues, so I thought I would ask what they have done. I have something workable right now, for my own use. What I am trying to arrive at is a clean and flexible scheme, to offers as patches to libevent. Regards, Steve From elliotf-libevent at gratuitous.net Mon Mar 5 18:33:30 2007 From: elliotf-libevent at gratuitous.net (Elliot F) Date: Mon Mar 5 18:40:57 2007 Subject: [Libevent-users] evhttp and reading large POSTed data (goal of a multipart/form-data parser) Message-ID: <45ECA8CA.5070806@gratuitous.net> Hello all, I'm writing a small web server using libevent's http layer, and would like some pointers on how to continue reading the input buffer. The end goal is a stream parser for a multipart/form-data. If one already exists, please let me know. When using 'EVBUFFER_DATA(evhttp_request->input_buffer)', I only get a portion of the upload. I'm assuming/hoping there is some way of associating a callback with that connection (or vice versa) to continue reading any data related to that request. Could someone point me in the right direction? Any tips and/or advice are welcome, including pointers to example code. Thank you in advance, Elliot F From provos at citi.umich.edu Mon Mar 5 22:47:33 2007 From: provos at citi.umich.edu (Niels Provos) Date: Mon Mar 5 22:47:36 2007 Subject: [Libevent-users] evhttp and reading large POSTed data (goal of a multipart/form-data parser) In-Reply-To: <45ECA8CA.5070806@gratuitous.net> References: <45ECA8CA.5070806@gratuitous.net> Message-ID: <850f7cbe0703051947k481526d5v8ce365fe86a56ced@mail.gmail.com> It would be the easiest if you were to give some example code. There are two callbacks that libevent supports. One gives you all data at the end of the request and the other one gives you updates as chunks arrive. Niels. On 3/5/07, Elliot F wrote: > Hello all, > > I'm writing a small web server using libevent's http layer, and would > like some pointers on how to continue reading the input buffer. The end > goal is a stream parser for a multipart/form-data. If one already > exists, please let me know. > > When using 'EVBUFFER_DATA(evhttp_request->input_buffer)', I only get a > portion of the upload. I'm assuming/hoping there is some way of > associating a callback with that connection (or vice versa) to continue > reading any data related to that request. Could someone point me in the > right direction? Any tips and/or advice are welcome, including pointers > to example code. > > Thank you in advance, > > Elliot F > _______________________________________________ > Libevent-users mailing list > Libevent-users@monkeymail.org > http://monkey.org/mailman/listinfo/libevent-users > > From elliotf-libevent at gratuitous.net Tue Mar 6 00:10:35 2007 From: elliotf-libevent at gratuitous.net (Elliot F) Date: Tue Mar 6 00:10:40 2007 Subject: [Libevent-users] evhttp and reading large POSTed data (goal of a multipart/form-data parser) In-Reply-To: <850f7cbe0703051947k481526d5v8ce365fe86a56ced@mail.gmail.com> References: <45ECA8CA.5070806@gratuitous.net> <850f7cbe0703051947k481526d5v8ce365fe86a56ced@mail.gmail.com> Message-ID: <45ECF7CB.8010600@gratuitous.net> Niels Provos wrote: > It would be the easiest if you were to give some example code. There > are two callbacks that libevent supports. One gives you all data at > the end of the request and the other one gives you updates as chunks > arrive. Of course code examples would help. Excuse me. I've attached a trimmed-down portion of code. I would be more interested in the "updates as chunks" as there is the possibility that the POSTed data would be very large. I was hoping/thinking that there was a callback that would be triggered on socket activity. If you could point me to what the two callbacks are, I would appreciate it as well as any other advice you might have on seeing my very (very) simple code that is early early on. > On 3/5/07, Elliot F wrote: >> Hello all, >> >> I'm writing a small web server using libevent's http layer, and would >> like some pointers on how to continue reading the input buffer. The end >> goal is a stream parser for a multipart/form-data. If one already >> exists, please let me know. >> >> When using 'EVBUFFER_DATA(evhttp_request->input_buffer)', I only get a >> portion of the upload. I'm assuming/hoping there is some way of >> associating a callback with that connection (or vice versa) to continue >> reading any data related to that request. Could someone point me in the >> right direction? Any tips and/or advice are welcome, including pointers >> to example code. >> >> Thank you in advance, >> >> Elliot F >> _______________________________________________ >> Libevent-users mailing list >> Libevent-users@monkeymail.org >> http://monkey.org/mailman/listinfo/libevent-users >> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: httpd.c Type: text/x-csrc Size: 1359 bytes Desc: not available Url : http://monkeymail.org/archives/libevent-users/attachments/20070305/01a5a30f/httpd.bin From elliotf-libevent at gratuitous.net Tue Mar 6 18:38:39 2007 From: elliotf-libevent at gratuitous.net (Elliot F) Date: Tue Mar 6 18:38:45 2007 Subject: [Libevent-users] evhttp and reading large POSTed data (goal of a multipart/form-data parser) In-Reply-To: <850f7cbe0703051947k481526d5v8ce365fe86a56ced@mail.gmail.com> References: <45ECA8CA.5070806@gratuitous.net> <850f7cbe0703051947k481526d5v8ce365fe86a56ced@mail.gmail.com> Message-ID: <45EDFB7F.7060504@gratuitous.net> Niels Provos wrote: > It would be the easiest if you were to give some example code. There > are two callbacks that libevent supports. One gives you all data at > the end of the request and the other one gives you updates as chunks > arrive. What are the two callbacks that libevent supports? I looked through the regression code and found the EVBUFFER_DATA function, but that seems to be the working buffer. When I used that, I would get the beginning portion of the POSTed data. I assume that is the method of accessing the working buffer. Do I have to replace the cb stored in the evbuffer struct that is stored in the evhttp_onnection struct? I would be more interested in the "updates as chunks arrive" callback, but would be interested both. Thank you in advance! > On 3/5/07, Elliot F wrote: >> Hello all, >> >> I'm writing a small web server using libevent's http layer, and would >> like some pointers on how to continue reading the input buffer. The end >> goal is a stream parser for a multipart/form-data. If one already >> exists, please let me know. >> >> When using 'EVBUFFER_DATA(evhttp_request->input_buffer)', I only get a >> portion of the upload. I'm assuming/hoping there is some way of >> associating a callback with that connection (or vice versa) to continue >> reading any data related to that request. Could someone point me in the >> right direction? Any tips and/or advice are welcome, including pointers >> to example code. >> >> Thank you in advance, >> >> Elliot F >> _______________________________________________ >> Libevent-users mailing list >> Libevent-users@monkeymail.org >> http://monkey.org/mailman/listinfo/libevent-users >> >> From ctrix+libevent at navynet.it Wed Mar 7 19:32:03 2007 From: ctrix+libevent at navynet.it (Massimo Cetra) Date: Wed Mar 7 19:32:29 2007 Subject: [Libevent-users] using threads and event_base... Is this an issue ? Message-ID: <45EF5983.1060201@navynet.it> Hi list, i'm experimenting with libevent in an heavy threaded application. The application works quite well using all the event_base_ stuff to handle the events in a thread safe way. My problem is that all the threads that i create have a short life (short as as little as a couple of minutes), so i have to event_init() once in every thread to have an event base struct. No problem about the methods... the problem arises because even_init() calls evsignal_init() which opens a socket pair which is completely lost and those sockets are never closed. I end up having a couple of fresh, unused and useless file descriptors for every thread that i have created and there seems no way to close those sockets and sockets are limited... Am i missing something or is this a known issue ? This behaviour makes libevent useless for many threaded apps ... Thanks for your answers... Massimo Cetra From sgrimm at facebook.com Wed Mar 7 20:27:41 2007 From: sgrimm at facebook.com (Steven Grimm) Date: Wed Mar 7 20:27:44 2007 Subject: [Libevent-users] using threads and event_base... Is this an issue ? In-Reply-To: <45EF5983.1060201@navynet.it> References: <45EF5983.1060201@navynet.it> Message-ID: <45EF668D.8020307@facebook.com> It looks like this would be a pretty easy problem to fix -- just make evsignal_init() check to see if the socketpair has already been allocated, and return without doing anything if so. Then you just need to make sure your first call to event_init() happens in only one thread. (I am assuming here that you don't actually need signal handling; if you do, then that first thread's event base would need to be kept around since it'd be the only one handling incoming signals.) Barring that, can you maintain a freelist of event bases and only allocate new ones when there aren't any unused ones left? That wouldn't fix the underlying bug, but would at least limit your file descriptor leakage to 2x the largest number of event bases in use at a time. -Steve Massimo Cetra wrote: > Hi list, > > i'm experimenting with libevent in an heavy threaded application. > > The application works quite well using all the event_base_ stuff to > handle the events in a thread safe way. > > My problem is that all the threads that i create have a short life > (short as as little as a couple of minutes), so i have to event_init() > once in every thread to have an event base struct. > > No problem about the methods... the problem arises because even_init() > calls evsignal_init() which opens a socket pair which is completely > lost and those sockets are never closed. > > I end up having a couple of fresh, unused and useless file descriptors > for every thread that i have created and there seems no way to close > those sockets and sockets are limited... > > Am i missing something or is this a known issue ? > > This behaviour makes libevent useless for many threaded apps ... > > Thanks for your answers... > > Massimo Cetra > > > _______________________________________________ > Libevent-users mailing list > Libevent-users@monkeymail.org > http://monkey.org/mailman/listinfo/libevent-users From ctrix+libevent at navynet.it Wed Mar 7 20:55:39 2007 From: ctrix+libevent at navynet.it (Massimo Cetra) Date: Wed Mar 7 20:56:04 2007 Subject: [Libevent-users] using threads and event_base... Is this an issue ? In-Reply-To: <45EF668D.8020307@facebook.com> References: <45EF5983.1060201@navynet.it> <45EF668D.8020307@facebook.com> Message-ID: <45EF6D1B.6070608@navynet.it> Steven Grimm wrote: > It looks like this would be a pretty easy problem to fix -- just make > evsignal_init() check to see if the socketpair has already been > allocated, and return without doing anything if so. yes. i have already sent a patch.. but the mail doesn't seems to have been accepted even if it was relayed and i had no response back. Maybe attachments are moderated. > Then you just need to make sure your first call to event_init() > happens in only one thread. (I am assuming here that you don't > actually need signal handling; if you do, then that first thread's > event base would need to be kept around since it'd be the only one > handling incoming signals.) > I call event_init once per thread and destroy it when the thread finishes. > Barring that, can you maintain a freelist of event bases and only > allocate new ones when there aren't any unused ones left? That > wouldn't fix the underlying bug, but would at least limit your file > descriptor leakage to 2x the largest number of event bases in use at a > time. the patch seems to solve the problem. I use 2 events per thread, i event_init() every thred and attach the events to that base. Then i delete the events and call event_base_free() [Note that this is not documented in the man page, afaik]. Even if experimental, now the support for threaded apps seems to work pretty well but i have not yet checked for memory leaks which willbe the 1st step after the application works. now the patch: ----------------------FIRSTLINE +++ libevent-1.3b.patched/signal.c 2007-03-08 02:01:54.000000000 +0100 @@ -57,6 +57,7 @@ static sig_atomic_t evsigcaught[NSIG]; volatile sig_atomic_t evsignal_caught = 0; +static int ev_signal_pair_done = 0; static struct event ev_signal; static int ev_signal_pair[2]; static int ev_signal_added; @@ -86,15 +87,17 @@ #define FD_CLOSEONEXEC(x) #endif -void -evsignal_init(void) +void evsignal_init(void) { + + if ( ev_signal_pair_done ) + return; /* * Our signal handler is going to write to one end of the socket * pair to wake up our event loop. The event loop then scans for * signals that got delivered. */ - if (socketpair(AF_UNIX, SOCK_STREAM, 0, ev_signal_pair) == -1) + if ( socketpair(AF_UNIX, SOCK_STREAM, 0, ev_signal_pair) == -1) event_err(1, "%s: socketpair", __func__); FD_CLOSEONEXEC(ev_signal_pair[0]); @@ -105,6 +108,8 @@ event_set(&ev_signal, ev_signal_pair[1], EV_READ, evsignal_cb, &ev_signal); ev_signal.ev_flags |= EVLIST_INTERNAL; + + ev_signal_pair_done = 1; } int -------------------------------LASTLINE From wouter at NLnetLabs.nl Thu Mar 8 03:29:51 2007 From: wouter at NLnetLabs.nl (Wouter Wijngaards) Date: Thu Mar 8 03:30:02 2007 Subject: [Libevent-users] using threads and event_base... Is this an issue ? In-Reply-To: <45EF6D1B.6070608@navynet.it> References: <45EF5983.1060201@navynet.it> <45EF668D.8020307@facebook.com> <45EF6D1B.6070608@navynet.it> Message-ID: <45EFC97F.6080101@nlnetlabs.nl> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Massimo Cetra wrote: > Steven Grimm wrote: >> It looks like this would be a pretty easy problem to fix -- just make >> evsignal_init() check to see if the socketpair has already been >> allocated, and return without doing anything if so. > yes. i have already sent a patch.. but the mail doesn't seems to have > been accepted even if it was relayed and i had no response back. Maybe > attachments are moderated. > Yes this is a problem. Also signal handling suffers from thread-unsafety. As well as event_loop_exit which is thread-unsafe and event_init() itself which is thread unsafe. Anyway, I have also sent a patch to Niels (last week), which solves your problem and all other problems. It also includes regression tests. He did note that he received the message. My solution opens the two file descriptors for every event_base, but also closes them on event_base_free. So that every event_base is capable of handling signals. So, my solution also solves your problem, I think. If only one thread handles signals, this will work. But also it is possible to have multiple event_bases in one thread and switch around which one you use, that one will handle signals for you :-) Best regards, Wouter -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFF78l/kDLqNwOhpPgRAjQcAKCDbZ0H149htnxqzXMUNk+iZyUNQwCgtqK5 WCGUgv3Sg3rzTchGQQAMUNE= =O2Fd -----END PGP SIGNATURE----- From ctrix+libevent at navynet.it Thu Mar 8 04:42:16 2007 From: ctrix+libevent at navynet.it (Massimo Cetra) Date: Thu Mar 8 04:42:38 2007 Subject: [Libevent-users] using threads and event_base... Is this an issue ? In-Reply-To: <45EFC97F.6080101@nlnetlabs.nl> References: <45EF5983.1060201@navynet.it> <45EF668D.8020307@facebook.com> <45EF6D1B.6070608@navynet.it> <45EFC97F.6080101@nlnetlabs.nl> Message-ID: <45EFDA78.5030606@navynet.it> Wouter Wijngaards wrote: > Also signal handling suffers from > thread-unsafety. As well as event_loop_exit which is thread-unsafe and > event_init() itself which is thread unsafe. Anyway, I have also sent a > patch to Niels (last week), which solves your problem and all other > problems. It also includes regression tests. He did note that he > received the message. > I hope your patch or a fix is included quickly... > My solution opens the two file descriptors for every event_base, but > also closes them on event_base_free. So that every event_base is capable > of handling signals. So, my solution also solves your problem, I think. > > I wonder if opening 2 FD per thread is necessary... Having 100 threads doing IO would need at least 300 sockets... Saving FD is not a bad idea... > If only one thread handles signals, this will work. But also it is > possible to have multiple event_bases in one thread and switch around > which one you use, that one will handle signals for you :-) > > This could be a good idea even if i have not yet the need of using multiple eventbases per thread... Thanks again, Cya, Max From wouter at NLnetLabs.nl Thu Mar 8 04:56:21 2007 From: wouter at NLnetLabs.nl (Wouter Wijngaards) Date: Thu Mar 8 05:01:56 2007 Subject: [Libevent-users] using threads and event_base... Is this an issue ? In-Reply-To: <45EFDA78.5030606@navynet.it> References: <45EF5983.1060201@navynet.it> <45EF668D.8020307@facebook.com> <45EF6D1B.6070608@navynet.it> <45EFC97F.6080101@nlnetlabs.nl> <45EFDA78.5030606@navynet.it> Message-ID: <45EFDDC5.9050900@nlnetlabs.nl> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Massimo Cetra wrote: > Wouter Wijngaards wrote: > I hope your patch or a fix is included quickly... Yes I need it too :) > I wonder if opening 2 FD per thread is necessary... > Having 100 threads doing IO would need at least 300 sockets... > Saving FD is not a bad idea... Hmm perhaps the opening of the pipe can be delayed until evsignal_add() is called? This will only open sockets for event bases where you use signal handling. >> > This could be a good idea even if i have not yet the need of using > multiple eventbases per thread... > > Thanks again, np. Wouter -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFF793FkDLqNwOhpPgRAoR3AKCvAXa/lY1exiW4oaKo8klMOvw5YwCePBqz otY3Nft0BYi4/4s5RA/ryOE= =a419 -----END PGP SIGNATURE----- From acd at weirdness.net Thu Mar 8 10:34:30 2007 From: acd at weirdness.net (Andrew Danforth) Date: Thu Mar 8 10:34:41 2007 Subject: [Libevent-users] using threads and event_base... Is this an issue ? In-Reply-To: <45EFDA78.5030606@navynet.it> References: <45EF5983.1060201@navynet.it> <45EF668D.8020307@facebook.com> <45EF6D1B.6070608@navynet.it> <45EFC97F.6080101@nlnetlabs.nl> <45EFDA78.5030606@navynet.it> Message-ID: What about creating a new initialization function that would take options via flags with one of them indicating that you don't want libevent to set up any signal handling stuff. We could name it something like event_init_advanced, or event_init_with_flags, or something like that. We would then need to update the libevent internals to skip evsignal_init if the "no signals" flag was present. With this approach we wouldn't break any existing code, as event_init would continue to function as it does today, and developers would have the flexibility to choose to use libevent signal handling in the future. I could whip up a patch for this if this seems like a good idea. Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://monkeymail.org/archives/libevent-users/attachments/20070308/9b64819e/attachment.htm From acd at weirdness.net Mon Mar 12 16:50:22 2007 From: acd at weirdness.net (Andrew Danforth) Date: Mon Mar 12 16:50:29 2007 Subject: [Libevent-users] Broken fixed width types in event.h Message-ID: Hi all, I primarily use libevent on Solaris. The libevent header file in latest release (1.3b) has nonstandard fixed width types that are not defined on Solaris. The C99 standard dictates the nomenclature uint32_t rather than u_int32_t and indicates that the fixed width types should be defined in stdint.h which Solaris provides. I'm not sure who is responsible for the new evtag_* stuff, but it would be nice to have it changed so that libevent will compile out of the box on Solaris again. Also, I'd like to echo some earlier comments where folks suggested that the additional functionality that is creeping in to libevent might best be kept in an external library. I don't use any of the HTTP, DNS, and now marshalling features, and it would be nice to not have to deal with new features breaking backwards compatibility. These additional interfaces could be distributed alongside the core libevent code and end users would have the option to use the additional library with higher level interfaces. Thanks, Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://monkeymail.org/archives/libevent-users/attachments/20070312/b0abc661/attachment-0001.htm From acd at weirdness.net Mon Mar 12 20:21:34 2007 From: acd at weirdness.net (Andrew Danforth) Date: Mon Mar 12 20:21:39 2007 Subject: [Libevent-users] sub-second timeouts broken on Solaris Message-ID: Hi all, The timercmp() macro provided by Solaris does not properly handle <= or >=. As a result, when timeout_next() is calculating how long to sleep and the next timeout occurs later in the current second, the code will call timerclear() and set the timeout to zero. This causes the main event loop to repeatedly call the dispatch function with a zero timeout until the timeout actually fires. I have included a patch which resolves the problem below. It should be fully backward compatible on systems with a working timercmp() -- I tested it on Linux. Another approach would be to update the autoconf stuff to check for the broken timercmp() and use the libevent built-in alternative. A --- libevent-1.3b.dist/event.c 2007-02-15 19:49:06.000000000 -0500 +++ libevent-1.3b/event.c 2007-03-12 18:30:30.568082270 -0400 @@ -726,7 +726,8 @@ if (gettime(&now) == -1) return (-1); - if (timercmp(&ev->ev_timeout, &now, <=)) { + /* Solaris timercmp macro does not properly handle <= */ + if (!timercmp(&ev->ev_timeout, &now, >)) { timerclear(tv); return (0); } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://monkeymail.org/archives/libevent-users/attachments/20070312/23bc4569/attachment.htm From liusifan at 21cn.com Wed Mar 14 21:34:31 2007 From: liusifan at 21cn.com (liusifan) Date: Wed Mar 14 21:34:46 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/Half-Async pattern Message-ID: <200703150934309214983@21cn.com> Hi, all SPServer is a server framework library written on C++ that implements the Half-Sync/Half-Async pattern ( http://www.cs.wustl.edu/~schmidt/PDF/HS-HA.pdf ). It's based on libevent in order to utilize the best I/O loop on any platform. SPServer can simplify TCP server construction. It is a hybrid system between threaded and event-driven, and exploits the advantages of both programming models. It exposes a threaded programming style to programmers, while simultaneously using event-driven style to process network connection. You can get the code from the SPServer home page: http://code.google.com/p/spserver/ http://spserver.googlecode.com/files/spserver-0.1.src.tar.gz stephen From william at 25thandClement.com Wed Mar 14 22:30:17 2007 From: william at 25thandClement.com (William Ahern) Date: Wed Mar 14 22:30:39 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/Half-Async pattern In-Reply-To: <200703150934309214983@21cn.com> References: <200703150934309214983@21cn.com> Message-ID: <20070315023017.GA24660@wilbur.25thandClement.com> On Thu, Mar 15, 2007 at 09:34:31AM +0800, liusifan wrote: > Hi, all > > SPServer is a server framework library written on C++ that implements the Half-Sync/Half-Async pattern ( http://www.cs.wustl.edu/~schmidt/PDF/HS-HA.pdf ). It's based on libevent in order to utilize the best I/O loop on any platform. > > SPServer can simplify TCP server construction. It is a hybrid system between threaded and event-driven, and exploits the advantages of both programming models. It exposes a threaded programming style to programmers, while simultaneously using event-driven style to process network connection. > > You can get the code from the SPServer home page: > http://code.google.com/p/spserver/ > http://spserver.googlecode.com/files/spserver-0.1.src.tar.gz > How is this different from GNU Pthreads? From william at 25thandClement.com Wed Mar 14 22:42:15 2007 From: william at 25thandClement.com (William Ahern) Date: Wed Mar 14 22:42:25 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/Half-Async pattern In-Reply-To: <20070315023017.GA24660@wilbur.25thandClement.com> References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> Message-ID: <20070315024215.GA16025@wilbur.25thandClement.com> On Wed, Mar 14, 2007 at 07:30:17PM -0700, William Ahern wrote: > On Thu, Mar 15, 2007 at 09:34:31AM +0800, liusifan wrote: > > Hi, all > > > > SPServer is a server framework library written on C++ that implements the Half-Sync/Half-Async pattern ( http://www.cs.wustl.edu/~schmidt/PDF/HS-HA.pdf ). It's based on libevent in order to utilize the best I/O loop on any platform. > > > > SPServer can simplify TCP server construction. It is a hybrid system between threaded and event-driven, and exploits the advantages of both programming models. It exposes a threaded programming style to programmers, while simultaneously using event-driven style to process network connection. > > > > You can get the code from the SPServer home page: > > http://code.google.com/p/spserver/ > > http://spserver.googlecode.com/files/spserver-0.1.src.tar.gz > > > > How is this different from GNU Pthreads? > RTFM ;) Seems like a lot of description for, what seems to me, is equivalent to this construct in my library: thread_run(&fn_to_run, fn_arg, &fn_callback, cb_context); Where fn_to_run() is called in a thread, and fn_callback() receives the return value of the function. From liusifan at 21cn.com Thu Mar 15 04:49:48 2007 From: liusifan at 21cn.com (liusifan) Date: Thu Mar 15 04:50:17 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/Half-Async patter References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> Message-ID: <200703151649481407698@21cn.com> Hi, William > >How is this different from GNU Pthreads? > Are you talking about http://www.gnu.org/software/pth/pth-manual.html ? As i know, Pth is another implementation of POSIX threads. SPServer is a server framework, it implements: 1.An Accepter for TCP Server 2.An asynchronous layer. It has only one thread ( event loop thread ), which is shared amongst all TCP connections. The event loop thread takes the responsibility to reading or writing data with TCP connections. In this layer, SPServer use libevent in order to utilize the best I/O loop on any platform. 3.A synchronous layer. It has multiple threads. Each activity/service can have its own thread of control. Long duration activities can run concurrently in their own threads without to worry about blocking other activities. 4.A queue between asynchronous layer and synchronous layer. The asynchronous layer reads data from TCP connections, and then puts the data and related handler to queue. The synchronous layer gets the data and related handler from the queue, and then wakes a thread to run the handler. The Acceptor/Asynchronous Layer/Queue are total implemented by SPServer framework. Using SPServer, programers only need to implement a subclass of SP_Handler and a subclass of SP_HandlerFactory. The subclass of SP_Handler will run in a independent thread, not in event loop thread. So programers can use synchronous programming style to implement the subclass of SP_Handler. Best regards, stephen From coroberti at gmail.com Thu Mar 15 06:12:09 2007 From: coroberti at gmail.com (Robert Iakobashvili) Date: Thu Mar 15 06:22:14 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/Half-Async patter In-Reply-To: <200703151649481407698@21cn.com> References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> <200703151649481407698@21cn.com> Message-ID: <7e63f56c0703150312n1f4f5e79u9be577600116706c@mail.gmail.com> Hi Stephen, On 3/15/07, liusifan wrote: > SPServer is a server framework, it implements: > 1.An Accepter for TCP Server > 2.An asynchronous layer. It has only one thread ( event loop thread ), > 3.A synchronous layer. It has multiple threads. > 4.A queue between asynchronous layer and synchronous layer. Very good job done. When the ACE-library is not an option: e.g. due to a large footprint (even without debugging info, optimized by size, and after cutting by so-reduce tool all unnecessaries it may reach up to 500 K) , it could be an interesting framework to consider. -- Sincerely, Robert Iakobashvili, coroberti %x40 gmail %x2e com ................................................................... Navigare necesse est, vivere non est necesse ................................................................... http://curl-loader.sourceforge.net A powerful open-source HTTP/S, FTP/S traffic generating, loading and testing tool. From liusifan at 21cn.com Thu Mar 15 06:36:08 2007 From: liusifan at 21cn.com (liusifan) Date: Thu Mar 15 06:36:28 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/ References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> <200703151649481407698@21cn.com> <7e63f56c0703150312n1f4f5e79u9be577600116706c@mail.gmail.com> Message-ID: <200703151836079062238@21cn.com> Hi, Robert > >Very good job done. > >When the ACE-library is not an option: e.g. due to a large footprint >(even without debugging info, optimized by size, and after cutting by >so-reduce tool all unnecessaries it may reach up to 500 K) , >it could be an interesting framework to consider. > Thanks for your kindness. I don't like to use ACE too. ACE is too large, too complex. So i decide to develop a lightweight Half-Sync/Half-Async framework. Best regards, stephen From Dmitry.Kalinin at kaspersky.com Thu Mar 15 06:47:21 2007 From: Dmitry.Kalinin at kaspersky.com (Dmitri Kalinin) Date: Thu Mar 15 06:45:20 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/ In-Reply-To: <200703151836079062238@21cn.com> References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> <200703151649481407698@21cn.com> <7e63f56c0703150312n1f4f5e79u9be577600116706c@mail.gmail.com> <200703151836079062238@21cn.com> Message-ID: <45F92439.7090004@kaspersky.com> Hi, Robert > >> Very good job done. >> >> When the ACE-library is not an option: e.g. due to a large footprint >> (even without debugging info, optimized by size, and after cutting by >> so-reduce tool all unnecessaries it may reach up to 500 K) , >> it could be an interesting framework to consider. >> >> > > Thanks for your kindness. > > I don't like to use ACE too. ACE is too large, too complex. > So i decide to develop a lightweight Half-Sync/Half-Async framework. > > Best regards, > stephen > > I wonder why was the design decision to use Half-Sync/Half-Async pattern but not the Leaders/Followers pattern which may be more efficient and scalable? (http://www.cs.wustl.edu/~schmidt/PDF/lf.pdf) Does libevent allows implementation of this multi-threaded pattern? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://monkeymail.org/archives/libevent-users/attachments/20070315/30ad8935/attachment-0001.htm From coroberti at gmail.com Thu Mar 15 07:04:38 2007 From: coroberti at gmail.com (Robert Iakobashvili) Date: Thu Mar 15 07:14:42 2007 Subject: [Libevent-users] A server framework library based on libevent that implements the Half-Sync/ In-Reply-To: <45F92439.7090004@kaspersky.com> References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> <200703151649481407698@21cn.com> <7e63f56c0703150312n1f4f5e79u9be577600116706c@mail.gmail.com> <200703151836079062238@21cn.com> <45F92439.7090004@kaspersky.com> Message-ID: <7e63f56c0703150404v94260d4g5a46ca1f3801be2f@mail.gmail.com> Dmitri, On 3/15/07, Dmitri Kalinin wrote: > Hi, Robert > I wonder why was the design decision to use Half-Sync/Half-Async pattern > but not the Leaders/Followers pattern which may be more efficient and > scalable? (http://www.cs.wustl.edu/~schmidt/PDF/lf.pdf) > Does libevent allows implementation of this multi-threaded pattern? Looks like a question to Stephen, not to me. Indeed Leaders/Followers has advantages in many cases, particularly when locking is expensive. However, I am not sure, that this is the right list to discuss it. -- Sincerely, Robert Iakobashvili, coroberti %x40 gmail %x2e com ................................................................... Navigare necesse est, vivere non est necesse ................................................................... http://curl-loader.sourceforge.net A powerful open-source HTTP/S, FTP/S traffic generating, loading and testing tool. From liusifan at 21cn.com Thu Mar 15 20:31:36 2007 From: liusifan at 21cn.com (liusifan) Date: Thu Mar 15 20:31:47 2007 Subject: [Libevent-users] A server framework library based on libeventthat implements the Half-Sync/ References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> <200703151649481407698@21cn.com> <7e63f56c0703150312n1f4f5e79u9be577600116706c@mail.gmail.com> <200703151836079062238@21cn.com> <45F92439.7090004@kaspersky.com> <7e63f56c0703150404v94260d4g5a46ca1f3801be2f@mail.gmail.com> Message-ID: <200703160831361713415@21cn.com> Hi, Dmitri >>>Dmitri, > >On 3/15/07, Dmitri Kalinin wrote: >> Hi, Robert >> I wonder why was the design decision to use Half-Sync/Half-Async pattern >> but not the Leaders/Followers pattern which may be more efficient and >> scalable? (http://www.cs.wustl.edu/~schmidt/PDF/lf.pdf) >> Does libevent allows implementation of this multi-threaded pattern? > Because I am familiar with Half-Sync/Half-Async pattern, so I choose this pattern. :) So it's an incidental choice. At first, i intend to develop an anti-spam SMTP proxy. With using ThreadPerConnection model, the proxy is working normal. But the number of the connections is limited by the number of the threads, so i try to find the other concurrent programming models. It looks Half-Sync/Half-Async is pretty fix for an anti-spam SMTP proxy. In the anti-spam SMTP proxy, these are many database operations. (1) Almost all database only provide blocking client API, and (2) it is difficult to program evening though we have non-blocking database client API. So i had to construct a hybrid framework between threaded and event-driven to simplify the construction of the anti-spam SMTP proxy. Best regards, Stephen From liusifan at 21cn.com Sun Mar 18 22:28:46 2007 From: liusifan at 21cn.com (liusifan) Date: Sun Mar 18 22:29:08 2007 Subject: [Libevent-users] A server framework library based on libeventthat implements the Half-Sync/ References: <200703150934309214983@21cn.com> <20070315023017.GA24660@wilbur.25thandClement.com> <200703151649481407698@21cn.com> <7e63f56c0703150312n1f4f5e79u9be577600116706c@mail.gmail.com> <200703151836079062238@21cn.com> <45F92439.7090004@kaspersky.com> <7e63f56c0703150404v94260d4g5a46ca1f3801be2f@mail.gmail.com> Message-ID: <200703191028455312420@21cn.com> Hi, all >On 3/15/07, Dmitri Kalinin wrote: >> Hi, Robert >> I wonder why was the design decision to use Half-Sync/Half-Async pattern >> but not the Leaders/Followers pattern which may be more efficient and >> scalable? (http://www.cs.wustl.edu/~schmidt/PDF/lf.pdf) >> Does libevent allows implementation of this multi-threaded pattern? The following is the comment of the ACE_TP_Reactor. This is ACE version 5.2, released Fri Oct 05 05:56:50 2001. /** * @file TP_Reactor.h * The (aka, Thread Pool Reactor) uses the * Leader/Followers pattern to demultiplex events among a pool of * threads. * ...... * TP_Reactor is a specialization of Select * Reactor to support thread-pool based event dispatching. This * Reactor takes advantage of the fact that events reported by *