libmicrohttpd
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [libmicrohttpd] MHD cannot write the entire chunked deflate data (de


From: silvioprog
Subject: Re: [libmicrohttpd] MHD cannot write the entire chunked deflate data (deflate/chunked)
Date: Sun, 10 Feb 2019 18:45:35 -0300

Hello Christian,

first of all, thanks a lot for answering, specially for the info regarding compress2(). After your message I decided to study this function internally and now I know its entire logic.

I've read the zlib manual and took a look at its examples/tests to get a better knowledge about deflate. And more, I found a logic in the MHD commit history which helped me a lot to understand it in practice. Lastly, I managed to solve the problem, and I wrote a tiny example for how compress data using the deflate algorithm and, if you agree, I could put the credits and the MHD licence on it and distribute it as example in the MHD sources.

This is the example (now it contains some error handling). Feel absolutely free to change it if you find something that could be improved, and let me know if it could be useful and distributed in the "src/examples/":

#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>
#include <zlib.h>
#include <microhttpd.h>

#define CHUNK 16384

struct Holder {
FILE *file;
z_stream stream;
void *buf;
};

static int
compressMem (z_stream *strm, const void *src, size_t src_size, size_t *offset, void **dest, size_t *dest_size,
void *tmp)
{
unsigned int have;
int ret;
int flush;
*dest = NULL;
*dest_size = 0;
do
{
if (src_size > CHUNK)
{
strm->avail_in = CHUNK;
src_size -= CHUNK;
flush = Z_NO_FLUSH;
}
else
{
strm->avail_in = (uInt) src_size;
flush = Z_SYNC_FLUSH;
}
*offset += strm->avail_in;
strm->next_in = (Bytef *) src;
do
{
strm->avail_out = CHUNK;
strm->next_out = tmp;
ret = deflate (strm, flush);
have = CHUNK - strm->avail_out;
*dest_size += have;
*dest = realloc (*dest, *dest_size);
if (NULL == *dest)
return MHD_NO;
memcpy ((*dest) + ((*dest_size) - have), tmp, have);
}
while (0 == strm->avail_out);
}
while (flush != Z_SYNC_FLUSH);
return (Z_OK == ret) ? MHD_YES : MHD_NO;
}

static ssize_t
read_cb (void *cls, uint64_t pos, char *mem, size_t size)
{
struct Holder *holder = cls;
void *src;
void *buf;
src = "" (size);
if (NULL == src)
return MHD_CONTENT_READER_END_WITH_ERROR;
size = fread (src, 1, size, holder->file);
if (size < 0)
{
size = MHD_CONTENT_READER_END_WITH_ERROR;
goto done;
}
if (0 == size)
{
size = MHD_CONTENT_READER_END_OF_STREAM;
goto done;
}
if (MHD_YES != compressMem (&holder->stream, src, size, &pos, &buf, &size, holder->buf))
size = MHD_CONTENT_READER_END_WITH_ERROR;
else
{
memcpy (mem, buf, size);
free (buf);
}
done:
free (src);
return size;
}

static void
free_cb (void *cls)
{
struct Holder *holder = cls;
fclose (holder->file);
deflateEnd (&holder->stream);
free (holder->buf);
free (holder);
}

static int
ahc_echo (void *cls, struct MHD_Connection *con, const char *url, const char *method, const char *version,
const char *upload_data, size_t *upload_size, void **ptr)
{
struct Holder *holder;
struct MHD_Response *res;
int ret;
(void) cls;
(void) url;
(void) method;
(void) version;
(void) upload_data;
(void) upload_size;
if (NULL == *ptr)
{
*ptr = (void *) 1;
return MHD_YES;
}
*ptr = NULL;
holder = calloc (1, sizeof (struct Holder));
if (!holder)
return MHD_NO;
holder->file = fopen (__FILE__, "rb");
if (NULL == holder->file)
goto error;
holder->buf = malloc (CHUNK);
if (NULL == holder->buf)
goto error;
ret = deflateInit(&holder->stream, Z_BEST_COMPRESSION);
if (ret != Z_OK)
goto error;
res = MHD_create_response_from_callback (MHD_SIZE_UNKNOWN, 1024, &read_cb, holder, &free_cb);
if (NULL == res)
goto error;
ret = MHD_add_response_header (res, MHD_HTTP_HEADER_CONTENT_ENCODING, "deflate");
if (MHD_YES != ret)
return MHD_NO;
ret = MHD_add_response_header (res, MHD_HTTP_HEADER_CONTENT_TYPE, "text/x-c");
if (MHD_YES != ret)
return MHD_NO;
ret = MHD_queue_response (con, MHD_HTTP_OK, res);
MHD_destroy_response (res);
return ret;
error:
fclose (holder->file);
free (holder->buf);
return MHD_NO;
}

int
main (int argc, char *const *argv)
{
struct MHD_Daemon *d;
unsigned int port;
if ((argc != 2) ||
(1 != sscanf (argv[1], "%u", &port)) ||
(UINT16_MAX < port))
{
fprintf (stderr,
"%s PORT\n", argv[0]);
return 1;
}
d = MHD_start_daemon (MHD_USE_AUTO | MHD_USE_INTERNAL_POLLING_THREAD, port, NULL, NULL,
&ahc_echo, NULL,
MHD_OPTION_END);
if (NULL == d)
return 1;
if (0 == port)
MHD_get_daemon_info (d, MHD_DAEMON_INFO_BIND_PORT, &port);
fprintf (stdout, "HTTP server running at http://localhost:%u\n\nPress ENTER to stop the server ...\n", port);
(void) getc (stdin);
MHD_stop_daemon (d);
return 0;
}

best,

On Sat, Feb 9, 2019 at 6:48 AM Christian Grothoff <address@hidden> wrote:
Dear Silvio,

The bug is actually in your logic. You are using libz wrong. First of
all, what you generate is "gzip", not "deflate" encoding. Second, if you
use compress2, you MUST supply the *entire* file at once, as it will
generate the gzip header and that must appear in the output only once.
The way you wrote your code, you are compressing the first 8k of the
file (part1.gz), then the next 8k of the file (part2.gz) and so on, and
then concatenating those part?.gz into one response.

cURL then detects that this is not a *single* well-formed gzip (after
part1.gz is done) and reports a "write error" (likely writing to the
deflate routine).  wget downloads the first 8k if you switch the header
to say 'gzip', and then bails.

The correct zlib API you must use to do the compression this way are the
deflateInit(), deflate() and deflateEnd() calls.  Between them, you must
udpate the z_stream structure, in particular avail_in, avail_out and
next_in and next_out, pointing to file data and the MHD buffer respectively.

Happy hacking!

Christian

On 2/9/19 5:06 AM, silvioprog wrote:
> Hello everyone,
>
> I've tried to compress a chunked data using ZLib & MHD, but the
> connection is closed while the data are transferred. Using the curl tool
> as client, it reports "curl: (23) Failed writing data" when we try to
> get a large file (about 100 kB). The attached MHD example uses the files
> "README" and "ChangeLog", available in the MHD sources, but it fails to
> send the "ChangeLog" file, because it is about 100 kB.
>
> It is easy to reproduce the problem. Just download the attached file, and:
>
> $ gcc -o httpsrv  main.c -lmicrohttpd -lz
> $ ./httpsrv
>
> Now, open another tab on your terminal, and:
>
> $ curl -v --compressed http://localhost:8080
>
> You will get something like:
>
> [snip]
> ...
> Tue Jul 23 11:32:00 MSK 2017
> Updated chunked_example.c to provide real illustration of usage of
> chunked encoding. -EG
>
> Thu Jul 13 21:41:00 MSK 2017
> Restored SIGPIPE suppression in TLS mode.
> * Failed writing data
> * stopped the pause stream!
> * Closing connection 0
> curl: (23) Failed writing data
> Added new value MHD_FEATURE_AUTOSUPPRESS
>
> Lastly, comment the line 56 (file = fopen(MHD_DIR "ChangeLog", "rb");)
> and uncomment the 57 (/*file = fopen(MHD_DIR "README", "rb");*/),
> rebuild the test and use curl again, you will get the the entire content
> of the README file.
>
> I've used the latest MHD from trunk (default build, by "./configure &&
> make install-strip"), and the ZLib version is 1.2.11. I'm on Xubuntu
> 18.04 64-bit.
>
> Could you confirm if it is some bug in MHD?
>
> Thanks in advance!
>
> P.S.: remember to change the line 7 (#define MHD_DIR "~/libmicrohttpd/")
> to avoid application crash, all the errors handling was omitted to make
> the example clear and small.
>
> --
> Silvio Clécio



--
Silvio Clécio

reply via email to

[Prev in Thread] Current Thread [Next in Thread]