Re: [libmicrohttpd] connection statistics

[Simon Newton]

On Sat, Feb 11, 2012 at 3:46 AM, Will Bryant <address@hidden> wrote:
> FWIW, Eivind isn't the only one who would use this - I needed it for my app 
> to produce app monitoring statistics.  (This sort of thing is a requirement 
> for devops - we only have one infrastructure component in production that is 
> not capable of instrumenting itself, and we're writing it out.)
>
> I ended up adding the following patch:
>
> unsigned int MHD_count_active_connections(struct MHD_Daemon *daemon) {
>        struct MHD_Connection *connection;
>        unsigned int result = 0;
>        if (pthread_mutex_lock(&daemon->cleanup_connection_mutex) != 0) return 
> 0;
>        connection = daemon->connections_head;
>        while (connection) { connection = connection->next; result++; }
>        if (pthread_mutex_unlock(&daemon->cleanup_connection_mutex) != 0) 
> return 0;
>        return result;
> }
>
> Although I don't like O(n) methods, for my app's use case this is acceptable. 
>  Of course, if I were rolling this patch into the library, I'd make it a 
> MHD_DaemonInfo option to MHD_get_daemon_info, but as a patch it's easier to 
> maintain as a separate method.
>
> Counting requests would be more complicated, but you can much more easily 
> count request starts and request completions using the provided callbacks, so 
> if that is an acceptable definition of active requests the difference between 
> those and the connection count can give you the number of inactive 
> connections.
>
> Yes, the OS can count active connections.  But I don't have to rewrite my 
> monitoring code and test harness code to work on three OSs with their own 
> netstat formats, so it's much easier to get it from the app itself.

+1. In my own app I support /debug which I use to dump all sorts of
internal state as key:value pairs. I have something that collects this
every few seconds and dumps the data into rrdtool to produce pretty
graphs.

Here is a small sample:

clients-connected: 1
cmd-line: -l3
fd-limit: 256
http_data_dir: /usr/local/share/olad/www
rpc-port: 9010
rpc-received: 9
rpc-send-errors: 0
rpc-sent: 9
server-uid: 7a70:c801a8c0
uptime-in-ms: 50927


Simon

>
>
> On 12/02/2012, at 00:20 , Christian Grothoff wrote:
>
>> The real issue with this kind of performance monitoring is that for 
>> multi-threaded applications it is very hard to give you a "sound" number.  
>> Also, I don't see how this would be helpful for the application that is 
>> running at the time.
>>
>> If the system operator wants this information, it is pretty easy to get a 
>> snapshot: lsof and netstat will show you how many connections the 
>> application has open and what the overall TCP state looks like for the OS.  
>> So for server status monitoring, why not use these standard tools?
>>
>> Now, if you actually have a very large number of mostly keep-alive 
>> connections, there are two possible answers: shorter timeout (so that they 
>> go away), or you're yet another person who'd like to see ePoll support for 
>> MHD (which is not that easy to add, hence it is not likely that I'll find 
>> the time to do this anytime soon).  Looking at how many select/poll calls 
>> you're doing and how long they take (for example, using 'strace -c -e 
>> trace=select,poll') might be a good idea here.
>>
>> In summary, I'm not sure I see a need for this kind of 
>> performance-monitoring support to be integrated with MHD, as the OS already 
>> gives you this and more high-quality information easily.
>>
>> My 2 cents!
>>
>> Happy hacking!
>>
>> Christian
>>
>> On 02/11/2012 12:40 AM, Eivind Sarto wrote:
>>> The completion callback handler is called whenever an HTTP request has 
>>> completed.
>>> At that point there is no way of knowing if the connection remains open.
>>>
>>> -eivind
>>> ________________________________________
>>> From: address@hidden address@hidden On Behalf Of Keith Mendoza 
>>> address@hidden
>>> Sent: Friday, February 10, 2012 3:32 PM
>>> To: libmicrohttpd development and user mailinglist
>>> Subject: Re: [libmicrohttpd] connection statistics
>>>
>>> If you're intent is to see the number of active connections at the
>>> moment I would say that this is something that you should be able to
>>> do from within your application. This might even prove useful to you
>>> at some future time in handling streaming of the video data. I can't
>>> provide any use cases, but I'm getting the "it might be useful" itch.
>>>
>>> If I understand it correctly, the completion handler is called when
>>> the HTTP connected between your application and the client is closed.
>>> So, if the keep-alive is still there wouldn't that mean that the
>>> connection is still open and the completion callback should not be
>>> called?
>>>
>>> On Fri, Feb 10, 2012 at 3:17 PM, Eivind Sarto<address@hidden>  wrote:
>>>> I am using libmicrohttpd for a video streaming project.
>>>> The HTTP server load can be quite high with a large number of requests per 
>>>> seconds and a high network bandwidth,
>>>> but the HTTP connections are mostly keep-alives.
>>>>
>>>> In order to troubleshoot load related problems, it can be difficult to get 
>>>> an idea of what is going on a given server.
>>>> I want to add some kind of server status monitoring and I would like to be 
>>>> able to display the number of active
>>>> connections/requests and total connections/requests (among other things).
>>>>
>>>> I can collect total and active number of requests from counters in the 
>>>> callback to the default handler and the completion handler.
>>>> However, I cannot find a way to get the total and active number of 
>>>> connections.  The only one who can keep an accurate count of
>>>> the number of connections is the internals of the library.
>>>>
>>>> Does anyone see any value to adding some statistics/counters to 
>>>> libmicrohttpd that can returned via an API?
>>>>
>>>> Or, is there something I am overlooking that I could do to display the 
>>>> number of active connections and total connections created?
>>>>
>>>> -eivind
>>>>
>>>>
>>>
>>>
>>
>>
>
>