libmicrohttpd
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [libmicrohttpd] epoll and memory leaks


From: Erik Smith
Subject: Re: [libmicrohttpd] epoll and memory leaks
Date: Fri, 10 Dec 2021 06:29:19 -0800

This is happening with the latest version (0.9.73) and on the master branch.

erik

On Thu, Dec 9, 2021 at 11:06 PM Evgeny Grin <k2k@yandex.ru> wrote:
Hi Erik,

It's hard to move forward without knowing exact MHD version used.

Please share information about your MHD version.

I suspect that your "Reply-To" header may confuse mailing list system.
Do not use "Reply-To" headers with mailing list.

--
Evgeny

-------- Original Message --------
From: Erik Smith <cruisercoder@gmail.com>
Sent: Friday, December 10, 2021, 03:29 UTC+3
Subject: [libmicrohttpd] epoll and memory leaks

> I've been able to reproduce this with a modified program from the
> examples to show the memory consumption I'm seeing.  I'm using jemalloc
> to capture memory consumption and here's what it looks like for the
> program below when repeatedly hitting the endpoint:
>
> allocated: 120072, active: 163840, resident: 9150464
> allocated: 150536, active: 196608, resident: 9228288
> allocated: 181000, active: 229376, resident: 9306112
> allocated: 211464, active: 262144, resident: 9383936
> allocated: 211464, active: 262144, resident: 9383936
> allocated: 241928, active: 294912, resident: 9461760
> allocated: 272392, active: 327680, resident: 9539584
> allocated: 272392, active: 327680, resident: 9539584
> allocated: 302856, active: 360448, resident: 9617408
>
> The delay in the handler and the use of ASAN tend to inflate the memory
> growth.   The key factor here seems to be the use of the thread poll
> with either poll or epoll.  Without the thread pool, there is no memory
> growth at all.  The growth happens on low connection rates (manual
> refreshing in the browser).   I haven't yet tried compiling MHD with ASAN.
>
> I'm also not getting responses to my threads in email for some reason
> but I'm checking the archive.
>
> #include <cstring>
> #include <iostream>
> #include <jemalloc/jemalloc.h>
> #include <microhttpd.h>
> #include <sstream>
> #include <thread>
>
> static enum MHD_Result handler(void *, struct MHD_Connection *connection,
>                                 const char *url, const char *method,
>                                 const char *, const char *, size_t *,
>                                 void **ptr) {
>    static int aptr;
>
>    if (&aptr != *ptr) {
>      *ptr = &aptr;
>      return MHD_YES;
>    }
>    *ptr = NULL;
>
>    std::this_thread::sleep_for(std::chrono::milliseconds(40));
>
>    size_t sz = sizeof(size_t);
>    uint64_t epoch = 1;
>    mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);
>    mallctl("epoch", &epoch, &sz, &epoch, sz);
>
>    std::size_t allocated, active, metadata, resident, mapped;
>    mallctl("stats.allocated", &allocated, &sz, NULL, 0);
>    mallctl("stats.active", &active, &sz, NULL, 0);
>    mallctl("stats.resident", &resident, &sz, NULL, 0);
>
>    std::stringstream s;
>    s << "allocated: " << allocated << ", active: " << active
>      << ", resident: " << resident << "\n";
>    auto msg = s.str();
>
>    std::cout << msg;
>
>    struct MHD_Response *response = MHD_create_response_from_buffer(
>        msg.size(), msg.data(), MHD_RESPMEM_MUST_COPY);
>    MHD_Result ret = MHD_queue_response(connection, MHD_HTTP_OK, response);
>    MHD_destroy_response(response);
>    return ret;
> }
>
> int main(int argc, char *argv[]) {
>    struct MHD_Daemon *d;
>
>    int port = argc > 1 ? atoi(argv[1]) : 10000;
>
>    // epoll mode with thread pool
>    unsigned int concurrency = std::thread::hardware_concurrency();
>    std::cout << "concurrency: " << concurrency << "\n";
>
>    d = MHD_start_daemon(MHD_USE_EPOLL_INTERNAL_THREAD |
> MHD_USE_ERROR_LOG, port,
>                         NULL, NULL, handler, NULL,
> MHD_OPTION_CONNECTION_TIMEOUT,
>                         (unsigned int)120, MHD_OPTION_STRICT_FOR_CLIENT,
> (int)1,
>                         MHD_OPTION_THREAD_POOL_SIZE, concurrency, NULL,
>                         MHD_OPTION_END);
>
>    if (d == NULL)
>      return 1;
>    std::cout << "listening on port: " << port << "\n";
>    std::cout << "hit key to stop"
>              << "\n";
>
>    // type a key to end
>    (void)getc(stdin);
>    MHD_stop_daemon(d);
>    return 0;
> }
>
>
>     Hi Erik,
>     Which MHD version are you using?
>     Some problems with externally added connections with epoll mode were
>     fixed in v0.9.72.
>     If you have any blocking calls, make sure that you use connection
>     suspend/resume. Alternatively, you can you use thread-per-connection
>     mode, this is less efficient, but simpler to implement.
>     epoll mode does not have special memory allocation, connections are
>     processed in the same way, like in other modes. MHD typically does
>     not allocate memory during connection processing, except when new
>     connection is started.
>     Do you use postprosessor or authentication functions? MHD has some
>     memory allocs in these functions.
>     The issue is not connected with quoted comment definitely. It is
>     just a bad wording. Actually nothing is leaked, but may be locked
>     until end of sending of response. Moreover, MHD does not use memory
>     pool in the way where such lock is possible. Memory pool is reset
>     after each request-reply cycle. Memory pool size for each connection
>     is fixed and cannot grow.
>     A few suggestions:
>     * make sure that you are using the latest MHD version (0.9.73 at the
>     moment), * check whether you destroy responses and free all
>     resources connected to responses, * if you are testing your code
>     with ASAN, make sure that leak detector is enabled. You can build
>     static MHD lib with ASAN and link it with our application compiled
>     with ASAN,
>     * use Valgrind or simpler tools like memstat or memprof.
>     --
>     Wishes,
>     Evgeny
>     On 22.11.2021 22:56, Erik Smith wrote:
>     /* Reallocate a block of memory obtained from the pool.
>     * This is particularly efficient when growing or
>     * shrinking the block that was last (re)allocated.
>     * If the given block is not the most recently
>     * (re)allocated block, the memory of the previous
>     * allocation may be leaked until the pool is
>     * destroyed or reset. */
>     Can anyone confirm whether this might be related?
>     ASAN does not seem to detect any issues in our code presently (not
>     sure about MHD)
>
>     We have started to experiment with running MHD with epoll + thread
>     pool as we do the FD limit in certain situations.  We understand
>     that there are caveats to this given that we have some
>     blocking database calls. This seems to get us past the FD limit
>     errors and the performance is similar.   However, we are running
>     into growing memory consumption in our server over time
>     running epoll+threads that require a restart frequently.   This does
>     not seem to occur with just epoll (without the thread pool).   We
>     are running jemalloc, but it does not seem to be related to the leak
>     when it is disabled.  There is the following comment in the MHD code
>     for the MHD_pool_reallocate function that might be connected to this
>     issue:
>

reply via email to

[Prev in Thread] Current Thread [Next in Thread]