Re: [libmicrohttpd] Problem with HTTP/1.1 chunked packets when connection is about to close

[Christian Grothoff]

On 4/27/21 10:41 AM, Iron Bug at gmail.com wrote:
> Greetings again.
> 
> But I take it, the chunked transfer is the only way for sending large files 
> that cannot/should not fit in buffer in RAM.

For large files you should use
MHD_create_response_from_fd_at_offset64().

That will do the right thing.

> Of course, I set unspecified size and intentionally use chunked transfer. 
> Moreover, I use pipelining for transferring many chunked files in one TCP 
> session and close it when client doesn't need it anymore to avoid dangling 
> TCP sessions that system may keep open for quite long time which is not 
> desireable in my case. I optimize my server part for memory print and it 
> should not read the whole files content into RAM. So chunked encoding is all 
> right and very useful.

Again, the above method will be MUCH better than you trying to manage
your memory footprint.

> The proper behavior (I specially checked in standard) when a client sends a 
> request with "Connection: close" header, the server should complete the 
> response (whether it's chunked or not) to the request and then close the 
> connection. And everything works fine until the "Connection: close" request 
> comes: then server misses the "Transfer-Encoding: chunked" header for the 
> last response. As I take it, it only has to check that connection is not 
> closed yet, it should be closed at the end of transfer, after processing the 
> last client's request. So it's not a bug when application forcibly sets 
> chunked encoding when connection is closed, it's just a hack to fix the real 
> bug with lost header in chunked transfer that should be ended with closing 
> connection.

MHD does not use chunked transfer unless it is needed or someone asks
for it. Here, it is not needed, _even if_ you provide the reply
incrementally.

> Btw, if I write here, what do you think about HTTP/2? I found 
> libmicrohttpd-http2 on the net, but it was a bit outdated and a little buggy. 
> I fixed bugs and merged both http2 and the latest libmicrohttpd versions 
> together, so now I use the extended version that supports both HTTP/1.x and 
> HTTP/2. But I wonder if HTTP/2 (or HTTP/3 that is a draft yet but is already 
> implemented in main browsers) are going to be supported in the library?

Properly supporting HTTP/2 and HTTP/3 will take developer time (and thus
funding) that we currently do not have. Evgeny and I would like to see
it happen eventually -- alas primarily in the context of the
microhttpd2.h / improved v2.0-API effort.

Which itself is also stalled because we'd really need to find a huge
chunk of time to spend on this to make it happen.  So contributions are
in principle welcome -- alas it is a huge task that I doubt can be done
purely with volunteers in a timely fashion.

> Sincerely, Yana A. Kireyonok aka Iron Bug
> 
> On Mon, 26 Apr 2021 12:00:39 -0400
> libmicrohttpd-request@gnu.org wrote:
> 
>> Send libmicrohttpd mailing list submissions to
>>      libmicrohttpd@gnu.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>      https://lists.gnu.org/mailman/listinfo/libmicrohttpd
>> or, via email, send a message with subject or body 'help' to
>>      libmicrohttpd-request@gnu.org
>>
>> You can reach the person managing the list at
>>      libmicrohttpd-owner@gnu.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of libmicrohttpd digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Problem with HTTP/1.1 chunked packets when connection is
>>       about to close (Iron Bug at gmail.com)
>>    2. Re: Problem with HTTP/1.1 chunked packets when connection is
>>       about to close (Evgeny Grin)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sun, 25 Apr 2021 21:45:58 +0500
>> From: "Iron Bug at gmail.com" <death.iron.bug@gmail.com>
>> To: libmicrohttpd@gnu.org
>> Subject: [libmicrohttpd] Problem with HTTP/1.1 chunked packets when
>>      connection is about to close
>> Message-ID: <20210425214558.c6cd4b410fc806e70505d80e@gmail.com>
>> Content-Type: text/plain; charset=US-ASCII
>>
>> Greetings.
>>
>> I use libmicrohttpd in my project and I found one bug in a very specific 
>> situation with chunked messages.
>>
>> The scheme I use is keep-alive connection for transferring some files 
>> (chunked content, HTTP/1.1 pipelining). When transfer is over, connection is 
>> closed by client request.
>>
>> I create responses with MHD_create_response_from_callback and the HTTP 
>> header "Transfer-Encoding: chunked" is automatically added to packet 
>> headers. But when the transfer is over and the last file requested, client 
>> sends "Connection: close". Theoretically, by HTTP standard, the server 
>> should process the request and then close the socket. And it does so, it 
>> sends the chunked data and closes the connection. But in the response to the 
>> last request it loses the "Transfer-Encoding: chunked" header and the client 
>> fails to accept the wrong formatted HTTP packet. If I manually add 
>> "Transfer-Encoding: chunked" to the last HTTP response, it works fine. 
>>
>> I think the problem hides in this code piece:
>>
>>       file src/microhttpd/connection.c, line 1376, in function 
>> have_chunked_upload:
>> -----------------------------------------------------------------
>>       if ( (MHD_NO != keepalive_possible (connection)) &&
>>            (MHD_str_equal_caseless_ (MHD_HTTP_VERSION_1_1,
>>                                      connection->version) ) )
>>       {
>>         if (NULL == have_encoding)
>>         {
>>           must_add_chunked_encoding = true;
>>           connection->have_chunked_upload = true;
>>         }
>>      ...
>> -----------------------------------------------------------------
>>
>> The connection is marked as non keep-alive at the moment after "Connection: 
>> close" and the content is still chunked. But the flag is not set and then 
>> the error occurs.
>>
>> Sincerely, Yana A. Kireyonok aka Iron Bug
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 26 Apr 2021 16:42:20 +0300
>> From: Evgeny Grin <k2k@yandex.ru>
>> To: libmicrohttpd@gnu.org
>> Subject: Re: [libmicrohttpd] Problem with HTTP/1.1 chunked packets
>>      when connection is about to close
>> Message-ID: <67616383-2a38-5ddc-353c-696456da1629@yandex.ru>
>> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>>
>> Привет, Яна!
>>
>> Actually this behaviour is not a bug.
>> Chunked encoding produces binary larger output and makes heavier system 
>> and network load, so MHD tries to avoid chunked output if possible.
>>
>> If you provide response size as the first argument for 
>> MHD_create_response_from_callback() MHD will never use chunked encoding.
>>
>> If you provide "-1" (MHD_SIZE_UNKNOWN) as the size then MHD will follow 
>> the next logic:
>> * if connection is "keep-alive" then only choice is chunked encoding and 
>> chunked encoding is used,
>> * if connection have to be closed after sending response then end of 
>> response can be indicated by end of connection so "heavier" chunked 
>> encoding is not used.
>>
>> This logic most probably will be re-considered soon as latest HTTP/1.1 
>> specification forbids responses without both encoding and content-size 
>> (and ending connection without specifying content size is unreliable).
>>
>> However I discovered bug related to chunked encoding (when application 
>> forcibly sets "Transfer-Encoding: chunked") and "Connection: close". It 
>> will be fixed in the next MHD release.
>>
>> -- 
>> Wishes,
>> Evgeny
>>
>> On 25.04.2021 19:45, Iron Bug at gmail.com wrote:
>>> Greetings.
>>>
>>> I use libmicrohttpd in my project and I found one bug in a very specific 
>>> situation with chunked messages.
>>>
>>> The scheme I use is keep-alive connection for transferring some files 
>>> (chunked content, HTTP/1.1 pipelining). When transfer is over, connection 
>>> is closed by client request.
>>>
>>> I create responses with MHD_create_response_from_callback and the HTTP 
>>> header "Transfer-Encoding: chunked" is automatically added to packet 
>>> headers. But when the transfer is over and the last file requested, client 
>>> sends "Connection: close". Theoretically, by HTTP standard, the server 
>>> should process the request and then close the socket. And it does so, it 
>>> sends the chunked data and closes the connection. But in the response to 
>>> the last request it loses the "Transfer-Encoding: chunked" header and the 
>>> client fails to accept the wrong formatted HTTP packet. If I manually add 
>>> "Transfer-Encoding: chunked" to the last HTTP response, it works fine.
>>>
>>> I think the problem hides in this code piece:
>>>
>>>        file src/microhttpd/connection.c, line 1376, in function 
>>> have_chunked_upload:
>>> -----------------------------------------------------------------
>>>        if ( (MHD_NO != keepalive_possible (connection)) &&
>>>             (MHD_str_equal_caseless_ (MHD_HTTP_VERSION_1_1,
>>>                                       connection->version) ) )
>>>        {
>>>          if (NULL == have_encoding)
>>>          {
>>>            must_add_chunked_encoding = true;
>>>            connection->have_chunked_upload = true;
>>>          }
>>>     ...
>>> -----------------------------------------------------------------
>>>
>>> The connection is marked as non keep-alive at the moment after "Connection: 
>>> close" and the content is still chunked. But the flag is not set and 
>> then the error occurs.
>>>
>>> Sincerely, Yana A. Kireyonok aka Iron Bug
>>>
>>
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: OpenPGP_signature
>> Type: application/pgp-signature
>> Size: 840 bytes
>> Desc: OpenPGP digital signature
>> URL: 
>> <https://lists.gnu.org/archive/html/libmicrohttpd/attachments/20210426/955bfc0a/attachment.sig>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> libmicrohttpd mailing list
>> libmicrohttpd@gnu.org
>> https://lists.gnu.org/mailman/listinfo/libmicrohttpd
>>
>>
>> ------------------------------
>>
>> End of libmicrohttpd Digest, Vol 144, Issue 2
>> *********************************************
> 
>

signature.asc
Description: OpenPGP digital signature