[libmicrohttpd] Re: libmicrohttpd question

[Christian Grothoff]

Right, stupid typos. Fixed in SVN HEAD. -Christian

On Wednesday, November 17, 2010 04:13:07 pm Jared Cantwell wrote:
> Dear Christian,
> 
> This looks perfect, and I think an excellent addition to the API.  Thanks
> for implementing it!
> 
> One small thing I ran into is that we are using c++ and SIZE_MAX is not
> defined.  The #else branch that defines MHD_CONTENT_READER_END_OF_STREAM is
> slightly incorrect.  I believe it should look something like:
> 
> #ifdef SIZE_MAX
> #define MHD_CONTENT_READER_END_OF_STREAM SIZE_MAX
> #define MHD_CONTENT_READER_END_WITH_ERROR (SIZE_MAX - 1)
> #else
> #define MHD_CONTENT_READER_END_OF_STREAM ((size_t) -1LL)
> #define MHD_CONTENT_READER_END_WITH_ERROR (((size_t) -1LL) - 1)
> #endif
> 
> note the macro name change, and that change from -LL to -1LL.  I will test
> the change in depth in the coming days, and let you know if I run into any
> issues.  Thanks again, and I'll report back as I use this awesome new
> feature.
> 
> ~Jared
> 
> On Wed, Nov 17, 2010 at 6:26 AM, Christian Grothoff
> 
> <address@hidden>wrote:
> > Dear Jared,
> > 
> > I've thought about your arguments and found that it would be indeed very
> > difficult to implement such signaling by other means.  As a result, I've
> > now
> > added "MHD_CONTENT_READER_END_WITH_ERROR" (aka -2 as size_t) to the API
> > -- and
> > introduced "MHD_CONTENT_READER_END_OF_STREAM" as a new name for what used
> > to
> > be "-1" (for successful end-of-stream signalling).
> > 
> > These changes are in SVN 13718/13719.  Please let me know if this works
> > for you (most of the diff is the documentation, the actual new code is
> > like 4 lines).
> > 
> > Happy hacking,
> > 
> > Christian
> > 
> > On Sunday 14 November 2010 22:09:35 Jared Cantwell wrote:
> > > I agree that there should be a cleaner way to handle this.  Our goal is
> > > simply to provide a URL through which content can be downloaded, and we
> > > would like to not have to specify the client used (nor write our own).
> > > 
> > > I can see many reasons why a chunked response can fail.  For example,
> > 
> > data
> > 
> > > corruption or failed disks, both of which may only be detected once the
> > > response has started.  My understanding of the purpose behind chunked
> > > encoding in the first place is to not have to completely prepare the
> > > response ahead of time.  However, what if after partially preparing the
> > > response, an unrecoverable error occurs?  Based on my readings of the
> > 
> > RFC,
> > 
> > > there doesn't seem to be a standard way of handling this situation.
> > > 
> > > Given the reasonable possibility of this situation, does it make sense
> > 
> > for
> > 
> > > the API to provide a way to "abort" the response?  I think that closing
> > 
> > the
> > 
> > > connection is a nice way to signal the other end that a fatal error has
> > > occurred, don't you?  To handle it otherwise, the developer would need
> > > to control both the client and the server, which seems unnecessary to
> > > me.
> > > 
> > > I can understand if you don't wish to incorporate this into your API,
> > > but would you mind giving a few pointers on where the code would need
> > 
> > modified
> > 
> > > to make this happen (as a test run)?
> > > 
> > > Just a thought
> > > 
> > > Jared
> > > 
> > > On Sun, Nov 14, 2010 at 3:47 PM, Christian Grothoff
> > > 
> > > <address@hidden>wrote:
> > > > Well, this is not something a well-behaved HTTP server should ever
> > > > do,
> > 
> > so
> > 
> > > > I think the solution with the footers is cleaner (and we should
> > 
> > probably
> > 
> > > > limit
> > > > the API to "legal" HTTP behavior --- after all, if the response
> > > > construction
> > > > could realistically fail, the developer should take care of this case
> > 
> > in
> > 
> > > > a nice way).
> > > > 
> > > > My 2 cents
> > > > 
> > > > Christian
> > > > 
> > > > On Sunday, November 14, 2010 09:07:08 pm Jared Cantwell wrote:
> > > > > Thanks Christian!
> > > > > 
> > > > > I think this will help a lot.  Would it make sense to have the
> > 
> > ability
> > 
> > > > > to prematurely close the connection with some error code (i.e. by
> > > > > returning -2)?  This is similar to what the try_ready_normal_body()
> > > > > does.  I would imagine that any normal client would interpret this
> > > > > as a failed request since they would have never received the
> > > > > trailing '0\n\r'.
> > > > > 
> > > > > ~Jared
> > > > > 
> > > > > On Sun, Nov 14, 2010 at 2:49 PM, Christian Grothoff
> > > > > 
> > > > > <address@hidden>wrote:
> > > > > > Dear Jared,
> > > > > > 
> > > > > > I was just about to fire back a response saying: "just add a
> > 
> > footer",
> > 
> > > > but
> > > > 
> > > > > > then
> > > > > > I decided to look for the API for that and I found that -- while
> > 
> > 99%
> > 
> > > > > > of the support for this is there -- a tiny function to do so was
> > > > > > missing from the API.
> > > > > > 
> > > > > > So I'm happy to introduce: "MHD_add_response_footer"; the only
> > > > 
> > > > difference
> > > > 
> > > > > > to
> > > > > > 'MHD_add_response_header' being that the 'kind' is set to
> > > > > > MHD_FOOTER_KIND. The
> > > > > > new API is in SVN 13683.
> > > > > > 
> > > > > > This helps you in that you simply add a footer that signals your
> > > > > > client software that some kind of error happened; doing so inside
> > 
> > of
> > 
> > > > > > the response callback while also returning "-1" should do the
> > 
> > trick.
> > 
> > > > > > In fact, the recommended approach would actually be to always add
> > > > > > a cryptographic checksum (MD5, SHA-1) as a footer.  That way,
> > > > > > your
> > > > 
> > > > clients
> > > > 
> > > > > > will
> > > > > > also be better at detecting other transmission errors in your
> > > > > > gigabytes of data.  This is the best approach, and certainly also
> > > > > > how HTTP intended footers
> > > > > > to be used.
> > > > > > 
> > > > > > Naturally, this only works if you can teach your HTTP client to
> > > > > > respect footers.  If you cannot do that, I don't have a good
> > > > > > answer for you.
> > > > > > 
> > > > > > Happy hacking,
> > > > > > 
> > > > > > Christian
> > > > > > 
> > > > > > On Sunday, November 14, 2010 06:00:37 pm Jared Cantwell wrote:
> > > > > > > Hi Christian,
> > > > > > > 
> > > > > > > Thanks for a great piece of software-- it has helped me
> > > > > > > tremendously
> > > > 
> > > > to
> > > > 
> > > > > > > quickly implement a key feature for us.  I have a question
> > > > > > > about chunked uploads that I was hoping you could help me
> > > > > > > with. According to the documentation (and my reading of the
> > > > > > > code), when using MHD_ContentReaderCallback to return chunks
> > > > > > > of data (i.e. in response
> > > > 
> > > > to
> > > > 
> > > > > > > a GET request), there doesn't seem to be a way to indicate that
> > 
> > the
> > 
> > > > > > > chunked response failed.  Returning 0 will try again, -1 will
> > > > > > > assume the transfer completed successfully.  Is there a way to
> > > > > > > indicate to MHD that a chunked response failed?  The reason I
> > > > > > > ask is that we are transferring files on
> > > > > > 
> > > > > > the
> > > > > > 
> > > > > > > order of 10s of GBs, and we're getting this data from
> > > > > > > throughout our cluster.  Midway through the transfer, anything
> > > > > > > could happen, and it
> > > > > > 
> > > > > > would
> > > > > > 
> > > > > > > be nice to just prematurely close the connection to signal
> > 
> > failure
> > 
> > > > > > (instead
> > > > > > 
> > > > > > > of assuming the end of the chunked data was reached).  Am I
> > 
> > missing
> > 
> > > > > > > something in the interface, or is this just not currently
> > 
> > possible?
> > 
> > > > > > > Thanks for any help!!
> > > > > > > ~Jared