[libmicrohttpd] Re: libmicrohttpd question

[Jared Cantwell]

Put more succinctly, this case seems identical to the case where the content length is specified, but -1 is returned before supplying all the data.  The only difference is that in this case, we were lucky enough to have known the content length before hand.  However, just because the content length is not known in the beginning, doesn't mean that prematurely returning -1 is any less valid of an option.

Jared

On Sun, Nov 14, 2010 at 4:09 PM, Jared Cantwell <address@hidden> wrote:

I agree that there should be a cleaner way to handle this.  Our goal is simply to provide a URL through which content can be downloaded, and we would like to not have to specify the client used (nor write our own).

I can see many reasons why a chunked response can fail.  For example, data corruption or failed disks, both of which may only be detected once the response has started.  My understanding of the purpose behind chunked encoding in the first place is to not have to completely prepare the response ahead of time.  However, what if after partially preparing the response, an unrecoverable error occurs?  Based on my readings of the RFC, there doesn't seem to be a standard way of handling this situation. 

Given the reasonable possibility of this situation, does it make sense for the API to provide a way to "abort" the response?  I think that closing the connection is a nice way to signal the other end that a fatal error has occurred, don't you?  To handle it otherwise, the developer would need to control both the client and the server, which seems unnecessary to me. 

I can understand if you don't wish to incorporate this into your API, but would you mind giving a few pointers on where the code would need modified to make this happen (as a test run)?

Just a thought

Jared

On Sun, Nov 14, 2010 at 3:47 PM, Christian Grothoff <address@hidden> wrote:

Well, this is not something a well-behaved HTTP server should ever do, so I
think the solution with the footers is cleaner (and we should probably limit
the API to "legal" HTTP behavior --- after all, if the response construction
could realistically fail, the developer should take care of this case in a
nice way).

My 2 cents

Christian

On Sunday, November 14, 2010 09:07:08 pm Jared Cantwell wrote:
> Thanks Christian!
>
> I think this will help a lot.  Would it make sense to have the ability to
> prematurely close the connection with some error code (i.e. by returning
> -2)?  This is similar to what the try_ready_normal_body() does.  I would
> imagine that any normal client would interpret this as a failed request
> since they would have never received the trailing '0\n\r'.
>
> ~Jared
>
> On Sun, Nov 14, 2010 at 2:49 PM, Christian Grothoff
>
> <address@hidden>wrote:
> > Dear Jared,
> >
> > I was just about to fire back a response saying: "just add a footer", but
> > then
> > I decided to look for the API for that and I found that -- while 99% of
> > the support for this is there -- a tiny function to do so was missing
> > from the API.
> >
> > So I'm happy to introduce: "MHD_add_response_footer"; the only difference
> > to
> > 'MHD_add_response_header' being that the 'kind' is set to
> > MHD_FOOTER_KIND. The
> > new API is in SVN 13683.
> >
> > This helps you in that you simply add a footer that signals your client
> > software that some kind of error happened; doing so inside of the
> > response callback while also returning "-1" should do the trick.
> >
> > In fact, the recommended approach would actually be to always add a
> > cryptographic checksum (MD5, SHA-1) as a footer.  That way, your clients
> > will
> > also be better at detecting other transmission errors in your gigabytes
> > of data.  This is the best approach, and certainly also how HTTP
> > intended footers
> > to be used.
> >
> > Naturally, this only works if you can teach your HTTP client to respect
> > footers.  If you cannot do that, I don't have a good answer for you.
> >
> > Happy hacking,
> >
> > Christian
> >
> > On Sunday, November 14, 2010 06:00:37 pm Jared Cantwell wrote:
> > > Hi Christian,
> > >
> > > Thanks for a great piece of software-- it has helped me tremendously to
> > > quickly implement a key feature for us.  I have a question about
> > > chunked uploads that I was hoping you could help me with.  According
> > > to the documentation (and my reading of the code), when using
> > > MHD_ContentReaderCallback to return chunks of data (i.e. in response to
> > > a GET request), there doesn't seem to be a way to indicate that the
> > > chunked response failed.  Returning 0 will try again, -1 will assume
> > > the transfer completed successfully.  Is there a way to indicate to
> > > MHD that a chunked response failed?  The reason I ask is that we are
> > > transferring files on
> >
> > the
> >
> > > order of 10s of GBs, and we're getting this data from throughout our
> > > cluster.  Midway through the transfer, anything could happen, and it
> >
> > would
> >
> > > be nice to just prematurely close the connection to signal failure
> >
> > (instead
> >
> > > of assuming the end of the chunked data was reached).  Am I missing
> > > something in the interface, or is this just not currently possible?
> > >
> > > Thanks for any help!!
> > > ~Jared