Re: [libmicrohttpd] Zero copy MHD_create_response_from_callback

[Christian Grothoff]

Dear Dan,

Sounds like you should be using MHD_create_response_from_iovec(). That 
way, you can avoid the memcpy() entirely.

Happy hacking!

Christian

On 11/17/22 08:27, Dan Inve wrote:
> Hi,
>
> I'm using libmicrohttpd to respond to an HTTP request for a large amount of
> data (sometimes GBs) using external select
> and MHD_create_response_from_callback with a known size. The data is
> sourced from hardware via a kernel driver. When the HTTP request is
> received, data is requested from the hardware, and then transferred in 64MB
> chunks to buffers allocated by the driver. Inside of
> my MHD_ContentReaderCallback, I check if the data transfer by the driver is
> complete, and currently memcpy 'max' bytes of data at a time from the
> driver buffer into the 'char *buf' passed to the callback.
>
> I've attempted two methods when it comes to specifying block_size for
> MHD_create_response_from_callback.
> If I specify 64MB, the problem is that the memcpy takes too long and blocks
> my application loop. I've also tried specifying a small block size like
> 64KB. However, the response is slow and inefficient because it is only
> responding 64KB per loop, which is doing other things as well.
>
> Is there a zero-copy way to use MHD_create_response_from_callback that
> doesn't require memcpy'ing the data into buf in the callback? Perhaps by
> specifying a pointer to a buffer in the callback (similar to
> MHD_create_response_from_buffer)?
>
> Thanks!
> -Dan