Re: [libmicrohttpd] epoll and memory leaks

[Evgeny Grin]

Hi Erik,

Which MHD version are you using?
Some problems with externally added connections with epoll mode were 
fixed in v0.9.72.

If you have any blocking calls, make sure that you use connection 
suspend/resume. Alternatively, you can you use thread-per-connection 
mode, this is less efficient, but simpler to implement.

epoll mode does not have special memory allocation, connections are 
processed in the same way, like in other modes. MHD typically does not 
allocate memory during connection processing, except when new connection 
is started.

Do you use postprosessor or authentication functions? MHD has some 
memory allocs in these functions.

The issue is not connected with quoted comment definitely. It is just a 
bad wording. Actually nothing is leaked, but may be locked until end of 
sending of response. Moreover, MHD does not use memory pool in the way 
where such lock is possible.
Memory pool is reset after each request-reply cycle. Memory pool size 
for each connection is fixed and cannot grow.

A few suggestions:
* make sure that you are using the latest MHD version (0.9.73 at the 
moment),
* check whether you destroy responses and free all resources connected 
to responses,
* if you are testing your code with ASAN, make sure that leak detector 
is enabled. You can build static MHD lib with ASAN and link it with our 
application compiled with ASAN,
* use Valgrind or simpler tools like memstat or memprof.

--
Wishes,
Evgeny



On 22.11.2021 22:56, Erik Smith wrote:
> We have started to experiment with running MHD with epoll + thread pool 
> as we do the FD limit in certain situations.  We understand that there 
> are caveats to this given that we have some blocking database calls.  
> This seems to get us past the FD limit errors and the performance is 
> similar.   However, we are running into growing memory consumption in 
> our server over time running epoll+threads that require a restart 
> frequently.   This does not seem to occur with just epoll (without the 
> thread pool).   We are running jemalloc, but it does not seem to be 
> related to the leak when it is disabled.  There is the following comment 
> in the MHD code for the MHD_pool_reallocate function that might be 
> connected to this issue:
>
> /* Reallocate a block of memory obtained from the pool.
>
> * This is particularly efficient when growing or
>
> * shrinking the block that was last (re)allocated.
>
> * If the given block is not the most recently
>
> * (re)allocated block, the memory of the previous
>
> * allocation may be leaked until the pool is
>
> * destroyed or reset. */
>
>
> Can anyone confirm whether this might be related?
>
>
> ASAN does not seem to detect any issues in our code presently (not sure 
> about MHD)

OpenPGP_signature
Description: OpenPGP digital signature