Re: [libmicrohttpd] libmicrohttpd Digest, Vol 79, Issue 4

[Eugenio Perez]

Hi Christian!

The two "*.c" I attached causes the connection drop, at least in my
environment. However, you need to execute them with MHD version
0.9.44, and with no strlen() patch applied. You can modify the buffer
size of the client & the URL length with argv[1]. Do you need some
modification over them?

I can't explain the connection memory & content/url length relation
with the bug by myself, they are just my observations :). Maybe buffer
wrapping? MHD_connection_handle_idle frequency call?

Thanks and happy hacking!

2015-11-03 23:52 GMT+01:00  <address@hidden>:
> Send libmicrohttpd mailing list submissions to
>         address@hidden
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.gnu.org/mailman/listinfo/libmicrohttpd
> or, via email, send a message with subject or body 'help' to
>         address@hidden
>
> You can reach the person managing the list at
>         address@hidden
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of libmicrohttpd digest..."
>
>
> Today's Topics:
>
>    1. Re: How to determine the file size for an upload and how to
>       read complete query string (Christian Grothoff)
>    2. Re: Hanging connections (Phil Rosenberg)
>    3. Re: How to determine the file size for an upload and how to
>       read complete query string (Christian Grothoff)
>    4. Re: Hanging connections (Kenneth Mastro)
>    5. Re: Some issues with MHD (Christian Grothoff)
>    6. Re: Hanging connections (Christian Grothoff)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 03 Nov 2015 22:49:31 +0100
> From: Christian Grothoff <address@hidden>
> To: address@hidden
> Subject: Re: [libmicrohttpd] How to determine the file size for an
>         upload and how to read complete query string
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=utf-8
>
> Somehow mailman discarded this reply by Dirk Brinkmeier
> <address@hidden>, but fortunately it notified me:
>
> Hi Ken.
>
> I also used MHD_get_connection_values() with MHD_HEADER_KIND before to
> store the browser connection values in a map.
> So with MHD_GET_ARGUMENT_KIND I got all the things I need.
> Thank you for the hint!
>
> But regarding the file size, I saw that wget is able to read the correct
> file size on start of download.
> I tried to figure out how wget is determine the size, but I got lost in
> the code somehow.
>
> Any further ideas are very welcome.
>
> Regards,
> Dirk.
>
>
> On 11/03/2015 06:55 PM, Dirk Brinkmeier wrote:
>> Hi.
>>
>> I'm very excited in using libmicrohttpd.
>> It has helped me a lot to develop a web server enabled application!
>>
>> But I have 2 questions:
>>
>> The first is regarding the complete size of a POSTed upload (e.g. file
>> upload).
>> Currently I'm using the value delivered by:
>>
>> MHD_lookup_connection_value(connection, MHD_HEADER_KIND,
>> MHD_HTTP_HEADER_CONTENT_LENGTH)
>>
>> But this value always seems to be higher than the actual data that is sent.
>> Or in other words it is higher than the file size to be uploaded.
>> So I wait for *upload_data_size to be empty an then I assume that the
>> POST upload is finished.
>> But I can't verify the uploaded file size to the real file size.
>>
>> Is the any chance to estimate the correct file size for an upload?
>>
>>
>> The second question belongs to requests that contains query data.
>> For example:
>> https://localhost:8099/json/T_Personal.json?order=asc&limit=10&offset=0&_=1436290313274
>>
>>
>> I'm reading each parameter by calling MHD_lookup_connection_value:
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "order");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "offset");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "limit");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "search");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "_");
>>
>> This has the disadvantage that I always have to know the parameters that
>> I expect.
>> Is there a way to get the complete query string (for parsing it
>> afterwards in my application)?
>> In the example above "order=asc&limit=10&offset=0&_=1436290313274"
>>
>>
>> Hope you can give me some hints...
>> I also hope that I don't have missed some basic chapters in the
>> documentation ;-)
>>
>> Thanks a lot in advance,
>> Dirk.
>>
>>
>>
>>
>>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 3 Nov 2015 22:02:02 +0000
> From: Phil Rosenberg <address@hidden>
> To: libmicrohttpd development and user mailinglist
>         <address@hidden>
> Subject: Re: [libmicrohttpd] Hanging connections
> Message-ID:
>         <address@hidden>
> Content-Type: text/plain; charset=UTF-8
>
> Hi Ken
> Thanks for the advice. As you say, my callback is not being called AT
> ALL. Here is my start call
>
> m_daemon = MHD_start_daemon(
> MHD_USE_THREAD_PER_CONNECTION,
> port,
> NULL,
> NULL,
> &HttpDaemon::connectionCallback,
> this,
> MHD_OPTION_NOTIFY_COMPLETED, &HttpDaemon::completeCallback, NULL,
> //variables which allow post to be processed
> MHD_OPTION_END);
>
> Since posting I have tried MHD_USE_SELECT_INTERNALLY and this seems to
> be working fine. Processing each request is very fast so the multi
> threading I guess isn't needed.
> You may notice above that my callback is actually a static method of a
> C++ class, but as a static method is equivalent to a regular function
> I don't believe this is a problem and it has never been a problem for
> other libraries where I have used callbacks (e.g. libCURL).
>
> I had the same isuue with GET too.
>
> I had seen the info about being called twice and (at least I believe )
> I am dealing with that.
>
> I will have a look at firebug and try to work things out from that end
> when I get a chance and I will see if I can create a minimal sample
> too.
>
> Thanks for all the ideas. It's always great to find what seems like a
> keen and active group at the other end of an email list. Now that I am
> running with select internally MHD seems perfect for my needs.
>
> Phil
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 03 Nov 2015 23:03:16 +0100
> From: Christian Grothoff <address@hidden>
> To: address@hidden
> Subject: Re: [libmicrohttpd] How to determine the file size for an
>         upload and how to read complete query string
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=utf-8
>
> On 11/03/2015 06:55 PM, Dirk Brinkmeier wrote:
>> Hi.
>>
>> I'm very excited in using libmicrohttpd.
>> It has helped me a lot to develop a web server enabled application!
>>
>> But I have 2 questions:
>>
>> The first is regarding the complete size of a POSTed upload (e.g. file
>> upload).
>> Currently I'm using the value delivered by:
>>
>> MHD_lookup_connection_value(connection, MHD_HEADER_KIND,
>> MHD_HTTP_HEADER_CONTENT_LENGTH)
>>
>> But this value always seems to be higher than the actual data that is sent.
>> Or in other words it is higher than the file size to be uploaded.
>> So I wait for *upload_data_size to be empty an then I assume that the
>> POST upload is finished.
>
> Sure, that also works.
>
>> But I can't verify the uploaded file size to the real file size.
>>
>> Is the any chance to estimate the correct file size for an upload?
>
> Odd, could you have a broken client doing the upload?
>
>> The second question belongs to requests that contains query data.
>> For example:
>> https://localhost:8099/json/T_Personal.json?order=asc&limit=10&offset=0&_=1436290313274
>>
>>
>> I'm reading each parameter by calling MHD_lookup_connection_value:
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "order");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "offset");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "limit");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "search");
>> MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "_");
>>
>> This has the disadvantage that I always have to know the parameters that
>> I expect.
>> Is there a way to get the complete query string (for parsing it
>> afterwards in my application)?
>> In the example above "order=asc&limit=10&offset=0&_=1436290313274"
>
> You should probably use MHD_get_connection_values(), that way you don't
> have to parse the string yourself.
>
> Happy hacking!
>
> Christian
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 3 Nov 2015 17:33:50 -0500
> From: Kenneth Mastro <address@hidden>
> To: libmicrohttpd development and user mailinglist
>         <address@hidden>
> Subject: Re: [libmicrohttpd] Hanging connections
> Message-ID:
>         <address@hidden>
> Content-Type: text/plain; charset="utf-8"
>
> Hmm.  I don't know what the defaults are for 'thread per connection' mode.
> I use that mode and everything is fine, so unless it's a new bug... I kind
> of suspect there might be no defaults and you have to specify the
> parameters.  That is, you may need to tell MHD how many threads to
> create/maintain/allow.  Otherwise, I don't see anything wrong with your
> 'start_daemon' call.  That 'connectionCallback' function of yours really
> should be called.
>
> I know the 'use select internally' mode is working for you, but if you'd
> like to try to continue using 'thread per connection', you could try adding
> something like the following to your 'start_daemon' call:
>
> MHD_OPTION_CONNECTION_LIMIT,   // specifies that the next arg is the max
> number of simultaneous connections
> (unsigned int)10,             // the number of permitted simultaneous
> connections
> MHD_OPTION_CONNECTION_TIMEOUT, // specifies that the next arg is how long
> any given connection can live
> (unsigned int)60,              // the number of seconds connections are
> allowed to live (0 would allow them to live indefinitely)
>
>
> If you don't have any long-lived requests, 'use select internally' is
> likely a good choice anyway.  You just have to be mindful that you can't
> service multiple requests simultaneously (unless you use the suspend/resume
> functionality and/or a thread pool).
>
>
> Regarding the static method in the C++ class - I suspect you're right.
> That should work.  Personally, I'm having it call a C function that has
> been 'externed' (to avoid any chance of function name mangling) and then
> have that call back into my C++ class.  One way or the other, I need access
> to persistent data in my class, so a static method versus a C function
> aren't really all that different.
>
>
> Ken
>
> PS: Christian's the real hero on the mailing list (and for MHD,
> obviously).  I've been using MHD for about a year and a half now, and in
> that time he's always been very responsive to people's questions.  It's
> amazing how much that helps an open source project.
>
>
> On Tue, Nov 3, 2015 at 5:02 PM, Phil Rosenberg <address@hidden>
> wrote:
>
>> Hi Ken
>> Thanks for the advice. As you say, my callback is not being called AT
>> ALL. Here is my start call
>>
>> m_daemon = MHD_start_daemon(
>> MHD_USE_THREAD_PER_CONNECTION,
>> port,
>> NULL,
>> NULL,
>> &HttpDaemon::connectionCallback,
>> this,
>> MHD_OPTION_NOTIFY_COMPLETED, &HttpDaemon::completeCallback, NULL,
>> //variables which allow post to be processed
>> MHD_OPTION_END);
>>
>> Since posting I have tried MHD_USE_SELECT_INTERNALLY and this seems to
>> be working fine. Processing each request is very fast so the multi
>> threading I guess isn't needed.
>> You may notice above that my callback is actually a static method of a
>> C++ class, but as a static method is equivalent to a regular function
>> I don't believe this is a problem and it has never been a problem for
>> other libraries where I have used callbacks (e.g. libCURL).
>>
>> I had the same isuue with GET too.
>>
>> I had seen the info about being called twice and (at least I believe )
>> I am dealing with that.
>>
>> I will have a look at firebug and try to work things out from that end
>> when I get a chance and I will see if I can create a minimal sample
>> too.
>>
>> Thanks for all the ideas. It's always great to find what seems like a
>> keen and active group at the other end of an email list. Now that I am
>> running with select internally MHD seems perfect for my needs.
>>
>> Phil
>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> <http://lists.gnu.org/archive/html/libmicrohttpd/attachments/20151103/707da972/attachment.html>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 03 Nov 2015 23:46:30 +0100
> From: Christian Grothoff <address@hidden>
> To: address@hidden
> Subject: Re: [libmicrohttpd] Some issues with MHD
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=windows-1252
>
> Hi!
>
> Ok, so I've solved most of the mystery.  SVN 36568 was indeed wrong, the
> pool reallocation was correct before 0.9.45 and will be again right in
> 0.9.46 (to be uploaded ASAP).  I now put a comment as to why it was OK
> to begin with.
>
> Eugenio: your 'strlen(line)' change is also in 0.9.46, as the code
> didn't tolerate "extra" empty lines in the protocol, but it is clearly
> better to not drop a connection just because between two requests there
> is an unexpected empty line. (Especially since the spec maybe unclear
> with respect to how many \r\n one should put at the end of a POST
> with/without a footer).  So that makes MHD more tolerant, which is a
> good thing (and harmless).
>
> What I didn't quite figure out is why there is then still this
> buffer-shrinkage that motivated the SVN 36568-change to begin with: I
> went again over the code (now with more understanding), and couldn't
> find anything.  So for that, I would now really appreciate a test that
> demonstrates this.
>
> Anyway, happy hacking!
>
> Christian
>
> On 11/03/2015 07:08 PM, Eugenio Perez wrote:
>> Hi All.
>>
>> First of all, congrats for this excellent library! I think it does
>> have a clear interface and it's very usable.
>>
>> However, I have found two issues. I don't know if I'm using bad the
>> library, so please correct me if I'm wrong.
>>
>> I'm trying to send, using one connection, a lot of small (~2K) POST
>> messages (in the real case, JSON text). If I use not chunked encoding
>> messages (i.e., specifying Content-Length header), all is OK.
>>
>> However, I've found that if I try to send the messages in chunks, MHD
>> drops the connection, and the client needs to reconnect again. Testing
>> in deep, I've found that:
>>
>> * If I increase MHD_OPTION_CONNECTION_MEMORY_LIMIT, the connections
>> are dropped slower (i.e., the same connection last longer).
>> * If I send smaller messages that the sent in http-post.c, MHD does not drop.
>> * If I change URL length, drops rate changes:
>>   * ./http-post "http://10.0.70.175:8080/aaaaaaaaaaaaaaaaaaaaaaaaaaaa";
>> causes drops
>>   * ./http-post
>> "http://10.0.70.175:8080/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; no drops
>> * If I apply the patch attached, MHD does not drop my connection
>> anymore. However, I don't know if it affects some other MHD internals.
>>
>> All of this happens with MHD 0.9.44. I've seen that you just announced
>> 0.9.45. However, if I update is even worse: connections are being
>> dropped almost every packet or 2 packet! If I bisect, the commit that
>> introduce the issue is r36568 (with message "-fix assertion failure
>> from race on shutdown and buffer shrinkage with pipelining").
>>
>> I attach client and server I'm using to test situation. In order to
>> detect reconnection, I just use:
>>
>> tcpdump -nni any "port 8080 and tcp[tcpflags] & (tcp-syn) != 0"
>>
>> Where 8080 is the post I'm using to testing.
>>
>> Am I using the bad options? Is the patch valid to fix the issue? if
>> not, what could I do to keep debugging this?
>>
>> Thanks and regards.
>>
>
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 03 Nov 2015 23:52:15 +0100
> From: Christian Grothoff <address@hidden>
> To: address@hidden
> Subject: Re: [libmicrohttpd] Hanging connections
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset="utf-8"
>
> On 11/03/2015 11:02 PM, Phil Rosenberg wrote:
>> Hi Ken
>> Thanks for the advice. As you say, my callback is not being called AT
>> ALL. Here is my start call
>
> That is truly strange.  Not even once on the first request? Have you
> tried something like 'fprintf' in the callback and connecting to the
> port with TELNET and doing a "GET / http/1.1"-style request by hand?
>
> If the callback still isn't called even once, then double-check that you
> did actually call MHD_start_daemon().
>
>> m_daemon = MHD_start_daemon(
>> MHD_USE_THREAD_PER_CONNECTION,
>> port,
>> NULL,
>> NULL,
>> &HttpDaemon::connectionCallback,
>> this,
>> MHD_OPTION_NOTIFY_COMPLETED, &HttpDaemon::completeCallback, NULL,
>> //variables which allow post to be processed
>> MHD_OPTION_END);
>>
>> Since posting I have tried MHD_USE_SELECT_INTERNALLY and this seems to
>> be working fine. Processing each request is very fast so the multi
>> threading I guess isn't needed.
>
> Eh, it's actually typically slower and only useful if your callback is
> blocking and that can't be helped.
>
> Happy hacking!
>
> Christian
> p.s.: the fix in 0.9.46 probably won't help you, as it only impacts
> pipelined connections and not if you're not called at all.
>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: signature.asc
> Type: application/pgp-signature
> Size: 819 bytes
> Desc: OpenPGP digital signature
> URL: 
> <http://lists.gnu.org/archive/html/libmicrohttpd/attachments/20151103/28b22cec/attachment.pgp>
>
> ------------------------------
>
> _______________________________________________
> libmicrohttpd mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/libmicrohttpd
>
>
> End of libmicrohttpd Digest, Vol 79, Issue 4
> ********************************************