Re: [libmicrohttpd] MHD cannot write the entire chunked deflate data (deflate/chunked)

[Christian Grothoff]

Dear Silvio,

The bug is actually in your logic. You are using libz wrong. First of
all, what you generate is "gzip", not "deflate" encoding. Second, if you
use compress2, you MUST supply the *entire* file at once, as it will
generate the gzip header and that must appear in the output only once.
The way you wrote your code, you are compressing the first 8k of the
file (part1.gz), then the next 8k of the file (part2.gz) and so on, and
then concatenating those part?.gz into one response.

cURL then detects that this is not a *single* well-formed gzip (after
part1.gz is done) and reports a "write error" (likely writing to the
deflate routine).  wget downloads the first 8k if you switch the header
to say 'gzip', and then bails.

The correct zlib API you must use to do the compression this way are the
deflateInit(), deflate() and deflateEnd() calls.  Between them, you must
udpate the z_stream structure, in particular avail_in, avail_out and
next_in and next_out, pointing to file data and the MHD buffer respectively.

Happy hacking!

Christian

On 2/9/19 5:06 AM, silvioprog wrote:
> Hello everyone,
> 
> I've tried to compress a chunked data using ZLib & MHD, but the
> connection is closed while the data are transferred. Using the curl tool
> as client, it reports "curl: (23) Failed writing data" when we try to
> get a large file (about 100 kB). The attached MHD example uses the files
> "README" and "ChangeLog", available in the MHD sources, but it fails to
> send the "ChangeLog" file, because it is about 100 kB.
> 
> It is easy to reproduce the problem. Just download the attached file, and:
> 
> $ gcc -o httpsrv  main.c -lmicrohttpd -lz
> $ ./httpsrv
> 
> Now, open another tab on your terminal, and:
> 
> $ curl -v --compressed http://localhost:8080
> 
> You will get something like:
> 
> [snip]
> ...
> Tue Jul 23 11:32:00 MSK 2017
> Updated chunked_example.c to provide real illustration of usage of
> chunked encoding. -EG
> 
> Thu Jul 13 21:41:00 MSK 2017
> Restored SIGPIPE suppression in TLS mode.
> * Failed writing data
> * stopped the pause stream!
> * Closing connection 0
> curl: (23) Failed writing data
> Added new value MHD_FEATURE_AUTOSUPPRESS
> 
> Lastly, comment the line 56 (file = fopen(MHD_DIR "ChangeLog", "rb");)
> and uncomment the 57 (/*file = fopen(MHD_DIR "README", "rb");*/),
> rebuild the test and use curl again, you will get the the entire content
> of the README file.
> 
> I've used the latest MHD from trunk (default build, by "./configure &&
> make install-strip"), and the ZLib version is 1.2.11. I'm on Xubuntu
> 18.04 64-bit.
> 
> Could you confirm if it is some bug in MHD?
> 
> Thanks in advance!
> 
> P.S.: remember to change the line 7 (#define MHD_DIR "~/libmicrohttpd/")
> to avoid application crash, all the errors handling was omitted to make
> the example clear and small.
> 
> --
> Silvio Clécio

signature.asc
Description: OpenPGP digital signature