[Libevent-users] http input buffer deallocation problem

[Cezary Rzewuski]

Niels Provos wrote:
> On Mon, May 12, 2008 at 3:33 AM, Cezary Rzewuski
> <C.Rzewuski at elka.pw.edu.pl> wrote:
>   
>>  I suspect there may be two reasons, but I don't know if this how libevent
>> works:
>>  1) there is any priority in events processed in libevent, and buffer
>> deallocation remains in the tail of events queue
>>  2) may be there is any HTTP header that causes libevent to postpone closing
>> connection (which crawler uses but wget not).
>>     
>
> We would have to see your code to figure out what's going on.  For
> example, HTTP has persistent connections, i.e. connections that stay
> open after a request has been handled.  It would also be good to know
> which version of libevent you are using.
>
> Niels.
>
>   
Actually my code is very similar to spybye. However, I don't need sites 
to be cached or to wait for antivirus
scanning, so it's kind of simplified spybye. libevent functions are used 
in exactly the same way as in spybye.
I use libevent1.4.3-stable.

After more extensive debugging I suspect, that, in case of heavy 
workload, libevent just doesn't complete
all the requests that are scheduled. 
I can see that my proxy receives a request from the crawler, sends the 
request to a www server,
then receives response from the server. Then evhttp_send_reply function 
is called to return
the reply to the crawler.
But then, strangely, function evhttp_write is never called with argument 
of this particular connection. Consequently, evhttp_send_done function, 
which is responsible for freeing connection and request structures, is 
also never called.
This causes that evhttp_connection and evhttp_request structures are 
never freed.
The situation I've described happens really randomly and only in case of 
crawling (which means many requests).
When I use proxy with browser (and human is the one that interacts:)), 
everything looks fine.

Cezary Rzewuski