Re: [libmicrohttpd] life of the strings / variables in callback

[Christian Grothoff]

Hi!

After sleeping over this, I now think I was a bit wrong: it doesn't make
sense for the application to copy the data if MHD painfully tries to
always avoid it, especially if it is not going to move until the
connection is destroyed.

So a better answer is that we need to document precisely which pointers
the application can safely alias and not copy, and which pointers the
application must copy if it still needs them. That may constrain MHD
development in the future, but I couldn't come up with a scenario where
those constraints would actually be a problem.

Consequently, I'm adding the following section to the manual:

>>>
@section Validity of pointers

MHD will give applications access to its internal data structures
via pointers via arguments and return values from its API.  This
creates the question as to how long those pointers are assured to
stay valid.

Most MHD data structures are associated with the connection of an
HTTP client.  Thus, pointers associated with a connection are
typically valid until the connection is finished, at which point
MHD will call the @code{MHD_RequestCompletedCallback} if one is
registered.  Applications that have such a callback registered
may assume that keys and values from the
@code{MHD_KeyValueIterator}, return values from
@code{MHD_lookup_connection_value} and the @code{url},
@code{method} and @code{version} arguments to the
@code{MHD_AccessHandlerCallback} will remain valid until the
respective @code{MHD_RequestCompletedCallback} is invoked.

In contrast, the @code{upload_data} argument of
@code{MHD_RequestCompletedCallback} as well as all pointers
from the @code{MHD_PostDataIterator} are only valid for the
duration of the callback.

Pointers returned from @code{MHD_get_response_header} are
valid as long as the response itself is valid.
<<<

Note that accessing these pointers from other threads will of
course still require locking to ensure that there is no race
with the MHD_RequestCompletedCallback, and that thus making
a copy might in some cases still be a better idea.  But I now think we
shouldn't recommend making a copy as the default, but instead have
well-defined semantics and then leave the decision to copy (or not) to
the developer.

Happy hacking!

Christian


On 02/08/2015 02:03 AM, Christian Grothoff wrote:
> On 02/07/2015 07:00 PM, chaman bagga wrote:
>>
>> Hi, Can someone point out whether the variables are valid in the
>> MHD_AccessHandlerCallback even after returning from callback when
>> using Suspend / Resume? 
> 
> Arguments provided by MHD are generally only assured to be valid for the
> duration of the callback, at least we deliberately did not document
> anything stronger.
> 
> If for performance reasons you really cannot afford to copy, the
> following is true today: With the exception of the "upload_data", the
> values reachable from the MHD_AccessHandlerCallback won't change (so
> particular, url, method, version and the "connection values" (URL
> arguments)).
> 
> However, you do need to worry about them being destroyed when the
> connection is finished. While you can listen for that even using a
> "MHD_RequestCompletedCallback", a multi-threaded implementation will at
> that point have plenty of fun (deadlocks, races) informing other threads
> that might use those values of their demise.  That is likely to be a
> bigger performance and correctness nightmare than just copying.
> 
> 
> Summary: you MUST always copy upload_data, everything else SHOULD be
> safe until MHD_RequestCompletedCallback(), but relying on it is not
> encouraged, *especially* if you involve other threads.
> 
>> I'm running an http server with internal
>> Select + epoll on linux, with a thread pool consisting of 4 threads.
>> From MHD_AccessHandlerCallback,  I call suspend as my processing
>> happens asynchronously in a different task. It requires an LDAP
>> lookup. On getting a response on LDAP, I resume the connection and
>> enqueue the response from the callback. This is as described in
>> [libmicrohttpd] Trouble getting a response sent from a separate
>> worker thread (with external select)
>>
>> At present, I am copying all the strings (url, upload_data etc.) as
>> its not clear if it is safe to pass the pointers to other threads, or
>> if the pointers are invalid once the callback returns. Response is
>> appreciated. Chaman
>>
> 
> I hope this clarifies the situation!
> 
> 
> Happy hacking!
> 
> Christian
>

signature.asc
Description: OpenPGP digital signature