[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [libmicrohttpd] How to send Chunked messages

From: silvioprog
Subject: Re: [libmicrohttpd] How to send Chunked messages
Date: Sat, 1 Dec 2018 11:19:17 -0300

It takes me less time using fopen() to download big files, but, after reading your answer, I'm not sure if it is really related to fopen vs fd or to my test environment. Anyway, I did a small patch in MHD that performs better results using fd. If you agree with that, I finally could upgrade my library to use MHD_response_from_fd-style functions and abandon fopen definitely. Firstly, consider the following example (errors handling was omitted to make it clear):

#define FILENAME "Win7.iso" // about 4.9 GB
#define USE_FOPEN

static ssize_t fileread_cb(void *handle, uint64_t offset, char *buf, size_t size) {return fread(buf, 1, size, handle);}
static void filefree_cb(void *handle) {fclose(handle);}

static int ahc_echo(void *cls, struct MHD_Connection *con, const char *url, const char *verb, const char *ver,
                    const char *ud, size_t *uds, void **p) {
    struct MHD_Response *res;
    int ret, fd;
    struct stat64 buf;
    FILE *file;
    if (!*p) {
        *p = (void *) 1;
        return MHD_YES;
    *p = NULL;
#if defined(USE_FOPEN)
    file = fopen64(FILENAME, "rb");
    fd = fileno(file);
    fstat64(fd, &buf);
    res = MHD_create_response_from_callback((uint64_t) buf.st_size, 16 * 1024, fileread_cb, file, filefree_cb);
    fd = open64(FILENAME, O_RDONLY);
    fstat64(fd, &buf);
    res = MHD_create_response_from_fd64((uint64_t) buf.st_size, fd);
    ret = MHD_queue_response(con, MHD_HTTP_OK, res);
    return ret;

using the example above, the fopen saves 8 ~ 11 seconds (the total average time download is ~35 seconds). However, if you agree to add a new define in mhd_limits.h:

#define MHD_FD_BLOCK_SIZE 4 * 1024

so MHD_create_response_from_callback() could use it:

  response = MHD_create_response_from_callback (size,

and we could change MHD_FD_BLOCK_SIZE at build time to use a custom value in such environments.

What do you think?

On Fri, Nov 30, 2018 at 7:18 PM Christian Grothoff <address@hidden> wrote:
silvioprog: I don't understand why you'd use fopen() here instead of
just using the MHD_response_from_fd-style response generation. On
platforms where sendfile() is unavailable, MHD falls back to the usual
reading of the file into a buffer, so I don't quite see how you
re-implementing that logic helps. Sure, the fopen()-API might do some
additional buffering, but if you give MHD enough memory per connection,
you should get exactly the same effect.  So this just seems
unnecessarily convoluted (but I didn't try to understand the motivation
behind every branch in that code).

Happy hacking!


On 11/30/18 4:32 AM, silvioprog wrote:
> Hi Santos and Christian.
> Santos, the Christian's answer was complete and clarified me a lot too. ☺
> Christian, it seems sendfile() is not available in mingw and in the
> Google's NDK (Android), but my library must work at least on
> Windows, Linux, Raspbian and Android, so I've used the fopen() in this
> way: link
> <> (sorry
> for put a link, but the code is a little bit large). Do you recommend to
> use fopen() in those cases? I would like to use sendfile() because it is
> very fast, but I can't implement it for Windows.
> Thanks in advance for your help!
> On Wed, Nov 28, 2018 at 1:55 PM Christian Grothoff <address@hidden
> <mailto:address@hidden>> wrote:
>     Dear Santos,
>     MHD can send chunked response to the client. That said, this is not a
>     question of how large the file is. Especially, if it is simply a very
>     large file, MHD allows you to use sendfile() to avoid having a copy of
>     the data in userspace.  Chunked encoding is usually only used if the
>     application simply does not know the size of the response ahead of time.
>     Happy hacking!
>     Christian
>     On 11/28/18 4:09 PM, Santos Das wrote:
>     > Hi,
>     >
>     > How can MHD send the chunked message to the client ?   
>     >
>     > When a server receives a simple GET request, and the response is so
>     > large that it must be sent back using the "chunked" procedure.
>     >
>     > An example would be if a server received a file GET request, and the
>     > response is a large data file.    
>     >
>     > Can you please point me to some example on how this can be done
>     using MHD ?
>     >
>     > Thanks in advance.
>     >
>     > Regards, Santos
> --
> Silvio Clécio
Silvio Clécio

reply via email to

[Prev in Thread] Current Thread [Next in Thread]