libmicrohttpd
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [libmicrohttpd] epoll and memory leaks


From: Evgeny Grin
Subject: Re: [libmicrohttpd] epoll and memory leaks
Date: Fri, 10 Dec 2021 10:06:20 +0300
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.2

Hi Erik,

It's hard to move forward without knowing exact MHD version used.

Please share information about your MHD version.

I suspect that your "Reply-To" header may confuse mailing list system. Do not use "Reply-To" headers with mailing list.

--
Evgeny

-------- Original Message --------
From: Erik Smith <cruisercoder@gmail.com>
Sent: Friday, December 10, 2021, 03:29 UTC+3
Subject: [libmicrohttpd] epoll and memory leaks

I've been able to reproduce this with a modified program from the examples to show the memory consumption I'm seeing.  I'm using jemalloc to capture memory consumption and here's what it looks like for the program below when repeatedly hitting the endpoint:

allocated: 120072, active: 163840, resident: 9150464
allocated: 150536, active: 196608, resident: 9228288
allocated: 181000, active: 229376, resident: 9306112
allocated: 211464, active: 262144, resident: 9383936
allocated: 211464, active: 262144, resident: 9383936
allocated: 241928, active: 294912, resident: 9461760
allocated: 272392, active: 327680, resident: 9539584
allocated: 272392, active: 327680, resident: 9539584
allocated: 302856, active: 360448, resident: 9617408

The delay in the handler and the use of ASAN tend to inflate the memory growth.   The key factor here seems to be the use of the thread poll with either poll or epoll.  Without the thread pool, there is no memory growth at all.  The growth happens on low connection rates (manual refreshing in the browser).   I haven't yet tried compiling MHD with ASAN.

I'm also not getting responses to my threads in email for some reason but I'm checking the archive.

#include <cstring>
#include <iostream>
#include <jemalloc/jemalloc.h>
#include <microhttpd.h>
#include <sstream>
#include <thread>

static enum MHD_Result handler(void *, struct MHD_Connection *connection,
                                const char *url, const char *method,
                                const char *, const char *, size_t *,
                                void **ptr) {
   static int aptr;

   if (&aptr != *ptr) {
     *ptr = &aptr;
     return MHD_YES;
   }
   *ptr = NULL;

   std::this_thread::sleep_for(std::chrono::milliseconds(40));

   size_t sz = sizeof(size_t);
   uint64_t epoch = 1;
   mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);
   mallctl("epoch", &epoch, &sz, &epoch, sz);

   std::size_t allocated, active, metadata, resident, mapped;
   mallctl("stats.allocated", &allocated, &sz, NULL, 0);
   mallctl("stats.active", &active, &sz, NULL, 0);
   mallctl("stats.resident", &resident, &sz, NULL, 0);

   std::stringstream s;
   s << "allocated: " << allocated << ", active: " << active
     << ", resident: " << resident << "\n";
   auto msg = s.str();

   std::cout << msg;

   struct MHD_Response *response = MHD_create_response_from_buffer(
       msg.size(), msg.data(), MHD_RESPMEM_MUST_COPY);
   MHD_Result ret = MHD_queue_response(connection, MHD_HTTP_OK, response);
   MHD_destroy_response(response);
   return ret;
}

int main(int argc, char *argv[]) {
   struct MHD_Daemon *d;

   int port = argc > 1 ? atoi(argv[1]) : 10000;

   // epoll mode with thread pool
   unsigned int concurrency = std::thread::hardware_concurrency();
   std::cout << "concurrency: " << concurrency << "\n";

  d = MHD_start_daemon(MHD_USE_EPOLL_INTERNAL_THREAD | MHD_USE_ERROR_LOG, port,                        NULL, NULL, handler, NULL, MHD_OPTION_CONNECTION_TIMEOUT,                        (unsigned int)120, MHD_OPTION_STRICT_FOR_CLIENT, (int)1,
                        MHD_OPTION_THREAD_POOL_SIZE, concurrency, NULL,
                        MHD_OPTION_END);

   if (d == NULL)
     return 1;
   std::cout << "listening on port: " << port << "\n";
   std::cout << "hit key to stop"
             << "\n";

   // type a key to end
   (void)getc(stdin);
   MHD_stop_daemon(d);
   return 0;
}


    Hi Erik,
    Which MHD version are you using?
    Some problems with externally added connections with epoll mode were
    fixed in v0.9.72.
    If you have any blocking calls, make sure that you use connection
    suspend/resume. Alternatively, you can you use thread-per-connection
    mode, this is less efficient, but simpler to implement.
    epoll mode does not have special memory allocation, connections are
    processed in the same way, like in other modes. MHD typically does
    not allocate memory during connection processing, except when new
    connection is started.
    Do you use postprosessor or authentication functions? MHD has some
    memory allocs in these functions.
    The issue is not connected with quoted comment definitely. It is
    just a bad wording. Actually nothing is leaked, but may be locked
    until end of sending of response. Moreover, MHD does not use memory
    pool in the way where such lock is possible. Memory pool is reset
    after each request-reply cycle. Memory pool size for each connection
    is fixed and cannot grow.
    A few suggestions:
    * make sure that you are using the latest MHD version (0.9.73 at the
    moment), * check whether you destroy responses and free all
    resources connected to responses, * if you are testing your code
    with ASAN, make sure that leak detector is enabled. You can build
    static MHD lib with ASAN and link it with our application compiled
    with ASAN,
    * use Valgrind or simpler tools like memstat or memprof.
    --
    Wishes,
    Evgeny
    On 22.11.2021 22:56, Erik Smith wrote:
    /* Reallocate a block of memory obtained from the pool.
    * This is particularly efficient when growing or
    * shrinking the block that was last (re)allocated.
    * If the given block is not the most recently
    * (re)allocated block, the memory of the previous
    * allocation may be leaked until the pool is
    * destroyed or reset. */
    Can anyone confirm whether this might be related?
    ASAN does not seem to detect any issues in our code presently (not
    sure about MHD)

    We have started to experiment with running MHD with epoll + thread
    pool as we do the FD limit in certain situations.  We understand
    that there are caveats to this given that we have some
    blocking database calls. This seems to get us past the FD limit
    errors and the performance is similar.   However, we are running
    into growing memory consumption in our server over time
    running epoll+threads that require a restart frequently.   This does
    not seem to occur with just epoll (without the thread pool).   We
    are running jemalloc, but it does not seem to be related to the leak
    when it is disabled.  There is the following comment in the MHD code
    for the MHD_pool_reallocate function that might be connected to this
    issue:

Attachment: OpenPGP_signature
Description: OpenPGP digital signature


reply via email to

[Prev in Thread] Current Thread [Next in Thread]