Re: [libmicrohttpd] size MHD_SIZE_UNKNOWN should never be returned in header

[Brecht Sanders]

Hello Christian,
Thanks for the quick response.
I looked in my code to make sure I wasn't missing anything.
Can you tell me if the MHD_ContentReaderCallback function needs to return 0 or -1 on end of data?

Then I wrote the attached test program in order to have a testcase.
On Windows (32-bit, compiled with MinGW) it returns:

    HTTP/1.1 200 OK
    Content-Length: 4294967295
    Content-Type: text/plain
    Date: Thu, 01 Apr 2010 11:42:52 GMT
   
    Hello world!

Then I ran it on 32-bit Linux (Debian Lenny 5.0.3) and I got:

    HTTP/1.1 200 OK
    Transfer-Encoding: chunked
    Content-Type: text/plain
    Date: Thu, 01 Apr 2010 11:48:41 GMT
   
    Hello world!

So there is definitely a difference: not only Content-Length is returned but also Transfer-Encoding: chunked is missing.
Or is this due to Debian being on an older version of libmicrohttpd (0.3.1-1)?

I hope this helps in finding the source of the problem.
Regards
    Brecht

Christian Grothoff wrote:

Hi!

I've looked all over the code and I cannot find any size_t/uint64_t confusion 
that would explain this.  If you were using MHD_SIZE_UNKNOWN in conjunction 
with "MHD_create_response_from_data", bad things would happen (since that's 
not allowed by the API), but other than that I cannot see how this could be:
"total_size" is set ONLY in the "create_response_from_callback" and then 
compared directly with MHD_SIZE_UNKNOWN.  

If you have a testcase (not with IE but using libcurl as a client) that checks 
for this and can reproduce it (at least on 32-bit archs), that would be very 
helpful.   As it stands, I cannot reproduce this -- and your patch, as you 
mention, is certainly rather unclean (and would break 4 GB -1byte transfers).

Happy hacking!

Christian

On Thursday 01 April 2010 10:15:09 am Brecht Sanders wrote:
  

Hi,
I use libmicrohttpd on Windows (compiled under MSYS/MinGW).
I have been looking at why my dynamic pages (using
MHD_create_response_from_callback) would not display on Internet
Explorer (version 6) while they were fine on  Firefox.
Finally I found it: the header returned includes:
    Content-Length: 4294967295
This is 0xFFFFFFFF.
In the code I see:
    #define MHD_SIZE_UNKNOWN  ((uint64_t) -1LL)
which probably corresponds, assuming you use uint64_t for size
everywhere and don't mix with size_t.
My first thought to fix this was to not return the length if it is
MHD_SIZE_UNKNOWN.
However, on my platform the (MHD_SIZE_UNKNOWN !=
connection->response->total_size) comparison didn't seem to match.
I worked around it for now with the patch below using size_t typecasts.
However I suspect an underlying problem where size gets truncated from
uint64_t to a smaller type (maybe size_t) somewhere else.
So my patch is not the perfect fix in case you will have file larger
than 4G.
Regards
    Brecht Sanders


patch -ulb src/daemon/connection.c << EOF
--- src/daemon/connection.c  2010-03-11 13:34:20 +0100
+++ src/daemon/connection.c  2010-04-01 10:02:50 +0200
@@ -498,4 +498,5 @@
     }
-  else if (NULL == MHD_get_response_header (connection->response,
-
MHD_HTTP_HEADER_CONTENT_LENGTH))
+  else if ((NULL == MHD_get_response_header (connection->response,
+
MHD_HTTP_HEADER_CONTENT_LENGTH)) &&
+           ((size_t)MHD_SIZE_UNKNOWN !=
(size_t)connection->response->total_size))
     {
EOF

    

#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include <string.h>
#ifdef __WIN32__
  #include <ws2tcpip.h>
#endif
#include <microhttpd.h>
#include <curl/curl.h>

#define HTTP_PORT 8000
#define HTTP_URL "http://127.0.0.1:8000/";

#if MHD_VERSION < 0x00040000
int content_read_callback (void* cls, size_t pos, char* buf, int max)
#else
int content_read_callback (void* cls, uint64_t pos, char* buf, int max)
#endif
{
  const char* body = (char*)cls;
  size_t bodylen = strlen(body);
  if (pos >= bodylen)
    return -1;
  size_t len = ((size_t)max <= bodylen - pos ? max : bodylen - pos);
  memcpy(buf, body, len);
  return len;
}

void content_free_callback (void *cls)
{
}

int on_request_data (void *cls, struct MHD_Connection *connection, const char 
*url, const char *method, const char *version, const char *upload_data, 
unsigned int *upload_data_size, void **con_cls)
{
        //on first call create data structure
        struct MHD_Response* response = NULL;
        unsigned int status_code = MHD_HTTP_OK;
  //create response
  const char* body = "Hello world!\n";
        response = MHD_create_response_from_callback((size_t)-1, 2048, 
content_read_callback, (void*)body, content_free_callback);
        if (!response) {
                status_code = MHD_HTTP_NOT_FOUND;
                response = MHD_create_response_from_data(9, (void*)"Not found", 
MHD_NO, MHD_NO);
                MHD_add_response_header(response, MHD_HTTP_HEADER_CONTENT_TYPE, 
"text/html");
        }
        //add header
  MHD_add_response_header(response, MHD_HTTP_HEADER_CONTENT_TYPE, "text/plain");
        //send response
        int ret = MHD_queue_response(connection, status_code, response);
        MHD_destroy_response(response);
        return ret;
}

void on_request_completed (void* cls, struct MHD_Connection* connection, void** 
con_cls, enum MHD_RequestTerminationCode toe)
{
}

int main (int argc, char *const *argv)
{
  //start HTTP server
  struct MHD_Daemon* webserver;
  if ((webserver = MHD_start_daemon(MHD_USE_THREAD_PER_CONNECTION | 
MHD_USE_DEBUG | MHD_USE_PEDANTIC_CHECKS, HTTP_PORT, NULL, NULL, 
&on_request_data, NULL, MHD_OPTION_NOTIFY_COMPLETED, on_request_completed, 
NULL, MHD_OPTION_END)) == NULL) {
                fprintf(stderr, "Error starting webserver on port %i\n", 
(int)HTTP_PORT);
    return 1;
  }
  //HTTP client
  CURL *curl;
  CURLcode res;
  if ((curl = curl_easy_init()) != NULL) {
    curl_easy_setopt(curl, CURLOPT_URL, HTTP_URL);
    //curl_easy_setopt(curl, CURLOPT_HTTPHEADER, 1L);
    curl_easy_setopt(curl, CURLOPT_WRITEHEADER, stdout);
    curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L);
    res = curl_easy_perform(curl);
    curl_easy_cleanup(curl);
  }
  //stop HTTP server
  MHD_stop_daemon(webserver);
  return 0;
}