[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [libmicrohttpd] epoll and memory leaks
|
From: |
Evgeny Grin |
|
Subject: |
Re: [libmicrohttpd] epoll and memory leaks |
|
Date: |
Sat, 11 Dec 2021 22:52:36 +0300 |
|
User-agent: |
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.2 |
I spend some time today trying to find the root cause for this issue.
1. jemalloc doesn't work well together with ASAN. Probably because both
are intercepting memory management functions.
2. Memory allocation is increased when jemalloc is used with your
example (I've fixed some minor things in it). It happens with any
polling function (select, poll, epoll).
3. Memory allocation (with jemalloc) stops at some level after a few
iterations. It jumps, but always returns back to some level.
4. Stable memory allocation level (with jemalloc) increased with more
threads in the thread pool. More threads - longer stabilization and
higher stabilization level.
5. Stopping and starting again of MHD (without restarting the process)
results in reset of memory allocation.
6. There are not connection in cleanup list or in any other list of MHD
when MHD is idle.
7. The situation is the same, regardless whether TLS support and other
options are enabled or not during configure/build time.
Looks like jemalloc does not fully (for system) deallocate memory for
each thread. When every thread has served connections, the allocation
level stabilized and does not further increase (at least with your example).
Here is the fixed example code used for testing:
----------------------------------------------------------------------
#include <cstring>
#include <iostream>
#include <jemalloc/jemalloc.h>
#include <microhttpd.h>
#include <sstream>
#include <thread>
static enum MHD_Result handler(void *, struct MHD_Connection *connection,
const char *url, const char *method,
const char *, const char *, size_t *,
void **ptr) {
static int aptr;
if (&aptr != *ptr) {
*ptr = &aptr;
return MHD_YES;
}
*ptr = NULL;
std::this_thread::sleep_for(std::chrono::milliseconds(400));
size_t sz;
uint64_t epoch = 1;
mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);
sz = sizeof(epoch);
mallctl("epoch", &epoch, &sz, &epoch, sz);
std::size_t allocated, active, resident;
allocated = active = resident = 0;
sz = sizeof(allocated);
mallctl("stats.allocated", &allocated, &sz, NULL, 0);
sz = sizeof(active);
mallctl("stats.active", &active, &sz, NULL, 0);
sz = sizeof(resident);
mallctl("stats.resident", &resident, &sz, NULL, 0);
std::stringstream s;
s << "allocated: " << allocated << ", active: " << active
<< ", resident: " << resident << "\n";
auto msg = s.str();
std::cout << msg;
struct MHD_Response *response = MHD_create_response_from_buffer(
msg.size(), (void*)msg.data(), MHD_RESPMEM_MUST_COPY);
MHD_Result ret = MHD_queue_response(connection, MHD_HTTP_OK, response);
MHD_destroy_response(response);
return ret;
}
int main(int argc, char *argv[]) {
struct MHD_Daemon *d;
int port = argc > 1 ? atoi(argv[1]) : 10000;
// epoll mode with thread pool
unsigned int concurrency = 2; //std::thread::hardware_concurrency();
std::cout << "concurrency: " << concurrency << "\n";
do {
d = MHD_start_daemon(MHD_USE_EPOLL_INTERNAL_THREAD |
MHD_USE_ERROR_LOG, port,
NULL, NULL, handler, NULL,
MHD_OPTION_CONNECTION_TIMEOUT, (unsigned int)10,
MHD_OPTION_STRICT_FOR_CLIENT, (int)1,
MHD_OPTION_THREAD_POOL_SIZE,
concurrency,
MHD_OPTION_END);
if (d == NULL)
return 1;
std::cout << "listening on port: " << port << "\n";
std::cout << "hit return to restart"
<< "\n";
(void)getc(stdin);
MHD_stop_daemon(d);
} while (1);
return 0;
}
----------------------------------------------------------------------
Possible workarounds for you:
1. Do not use jemalloc.
2. Use thread-per-connection. The is less efficient, but end of thread
resets jemalloc allocations.
3. Try to use several MHD daemons with shared listening port. See
MHD_OPTION_LISTENING_ADDRESS_REUSE and MHD_OPTION_LISTEN_SOCKET.
4. Stop and restart MHD daemon to reset jemalloc allocation (ugly, but
works).
5. Combination of 3. and 4. Use several daemons and restart them by
schedule. Then you can provide continuous service without interruption.
See MHD_quiesce_daemon().
Maybe your problem is related to memory fragmentation. Intensive
allocation/deallocation with various small regions can be problematic.
Potentially it could be related to memory fragmentation handling by
jemalloc.
Hope this helps.
--
Evgeny
-------- Original Message --------
From: Erik Smith <cruisercoder@gmail.com>
Sent: Saturday, December 11, 2021, 21:52 UTC+3
Subject: [libmicrohttpd] epoll and memory leaks
Update: using jemalloc stats may not be useful without more
configuration as I think the increase might just be some dynamic
pre-allocation within jemalloc. When I remove the jemaloc dependency
and just monitor it with top, using top -p $(pidof demo_server), I see
no increase in RES but some increase in VIRT. There is something about
the MHD thread pool mode that is problematic for us that requires us to
restart our production servers after a few days. It does not appear to
happen when the thread pool is not used or potentially it is some
interaction with jemalloc.
Erik
On Fri, Dec 10, 2021 at 6:29 AM Erik Smith <cruisercoder@gmail.com
<mailto:cruisercoder@gmail.com>> wrote:
This is happening with the latest version (0.9.73) and on the master
branch.
erik
On Thu, Dec 9, 2021 at 11:06 PM Evgeny Grin <k2k@yandex.ru
<mailto:k2k@yandex.ru>> wrote:
Hi Erik,
It's hard to move forward without knowing exact MHD version used.
Please share information about your MHD version.
I suspect that your "Reply-To" header may confuse mailing list
system.
Do not use "Reply-To" headers with mailing list.
--
Evgeny
-------- Original Message --------
From: Erik Smith <cruisercoder@gmail.com
<mailto:cruisercoder@gmail.com>>
Sent: Friday, December 10, 2021, 03:29 UTC+3
Subject: [libmicrohttpd] epoll and memory leaks
> I've been able to reproduce this with a modified program from
the
> examples to show the memory consumption I'm seeing. I'm
using jemalloc
> to capture memory consumption and here's what it looks like
for the
> program below when repeatedly hitting the endpoint:
>
> allocated: 120072, active: 163840, resident: 9150464
> allocated: 150536, active: 196608, resident: 9228288
> allocated: 181000, active: 229376, resident: 9306112
> allocated: 211464, active: 262144, resident: 9383936
> allocated: 211464, active: 262144, resident: 9383936
> allocated: 241928, active: 294912, resident: 9461760
> allocated: 272392, active: 327680, resident: 9539584
> allocated: 272392, active: 327680, resident: 9539584
> allocated: 302856, active: 360448, resident: 9617408
>
> The delay in the handler and the use of ASAN tend to inflate
the memory
> growth. The key factor here seems to be the use of the
thread poll
> with either poll or epoll. Without the thread pool, there is
no memory
> growth at all. The growth happens on low connection rates
(manual
> refreshing in the browser). I haven't yet tried compiling
MHD with ASAN.
>
> I'm also not getting responses to my threads in email for
some reason
> but I'm checking the archive.
>
> #include <cstring>
> #include <iostream>
> #include <jemalloc/jemalloc.h>
> #include <microhttpd.h>
> #include <sstream>
> #include <thread>
>
> static enum MHD_Result handler(void *, struct MHD_Connection
*connection,
> const char *url, const char
*method,
> const char *, const char *,
size_t *,
> void **ptr) {
> static int aptr;
>
> if (&aptr != *ptr) {
> *ptr = &aptr;
> return MHD_YES;
> }
> *ptr = NULL;
>
> std::this_thread::sleep_for(std::chrono::milliseconds(40));
>
> size_t sz = sizeof(size_t);
> uint64_t epoch = 1;
> mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);
> mallctl("epoch", &epoch, &sz, &epoch, sz);
>
> std::size_t allocated, active, metadata, resident, mapped;
> mallctl("stats.allocated", &allocated, &sz, NULL, 0);
> mallctl("stats.active", &active, &sz, NULL, 0);
> mallctl("stats.resident", &resident, &sz, NULL, 0);
>
> std::stringstream s;
> s << "allocated: " << allocated << ", active: " << active
> << ", resident: " << resident << "\n";
> auto msg = s.str();
>
> std::cout << msg;
>
> struct MHD_Response *response =
MHD_create_response_from_buffer(
> msg.size(), msg.data(), MHD_RESPMEM_MUST_COPY);
> MHD_Result ret = MHD_queue_response(connection,
MHD_HTTP_OK, response);
> MHD_destroy_response(response);
> return ret;
> }
>
> int main(int argc, char *argv[]) {
> struct MHD_Daemon *d;
>
> int port = argc > 1 ? atoi(argv[1]) : 10000;
>
> // epoll mode with thread pool
> unsigned int concurrency =
std::thread::hardware_concurrency();
> std::cout << "concurrency: " << concurrency << "\n";
>
> d = MHD_start_daemon(MHD_USE_EPOLL_INTERNAL_THREAD |
> MHD_USE_ERROR_LOG, port,
> NULL, NULL, handler, NULL,
> MHD_OPTION_CONNECTION_TIMEOUT,
> (unsigned int)120,
MHD_OPTION_STRICT_FOR_CLIENT,
> (int)1,
> MHD_OPTION_THREAD_POOL_SIZE,
concurrency, NULL,
> MHD_OPTION_END);
>
> if (d == NULL)
> return 1;
> std::cout << "listening on port: " << port << "\n";
> std::cout << "hit key to stop"
> << "\n";
>
> // type a key to end
> (void)getc(stdin);
> MHD_stop_daemon(d);
> return 0;
> }
>
>
> Hi Erik,
> Which MHD version are you using?
> Some problems with externally added connections with
epoll mode were
> fixed in v0.9.72.
> If you have any blocking calls, make sure that you use
connection
> suspend/resume. Alternatively, you can you use
thread-per-connection
> mode, this is less efficient, but simpler to implement.
> epoll mode does not have special memory allocation,
connections are
> processed in the same way, like in other modes. MHD
typically does
> not allocate memory during connection processing, except
when new
> connection is started.
> Do you use postprosessor or authentication functions? MHD
has some
> memory allocs in these functions.
> The issue is not connected with quoted comment
definitely. It is
> just a bad wording. Actually nothing is leaked, but may
be locked
> until end of sending of response. Moreover, MHD does not
use memory
> pool in the way where such lock is possible. Memory pool
is reset
> after each request-reply cycle. Memory pool size for each
connection
> is fixed and cannot grow.
> A few suggestions:
> * make sure that you are using the latest MHD version
(0.9.73 at the
> moment), * check whether you destroy responses and free all
> resources connected to responses, * if you are testing
your code
> with ASAN, make sure that leak detector is enabled. You
can build
> static MHD lib with ASAN and link it with our application
compiled
> with ASAN,
> * use Valgrind or simpler tools like memstat or memprof.
> --
> Wishes,
> Evgeny
> On 22.11.2021 22:56, Erik Smith wrote:
> /* Reallocate a block of memory obtained from the pool.
> * This is particularly efficient when growing or
> * shrinking the block that was last (re)allocated.
> * If the given block is not the most recently
> * (re)allocated block, the memory of the previous
> * allocation may be leaked until the pool is
> * destroyed or reset. */
> Can anyone confirm whether this might be related?
> ASAN does not seem to detect any issues in our code
presently (not
> sure about MHD)
>
> We have started to experiment with running MHD with
epoll + thread
> pool as we do the FD limit in certain situations. We
understand
> that there are caveats to this given that we have some
> blocking database calls. This seems to get us past the FD
limit
> errors and the performance is similar. However, we are
running
> into growing memory consumption in our server over time
> running epoll+threads that require a restart frequently.
This does
> not seem to occur with just epoll (without the thread
pool). We
> are running jemalloc, but it does not seem to be related
to the leak
> when it is disabled. There is the following comment in
the MHD code
> for the MHD_pool_reallocate function that might be
connected to this
> issue:
>
OpenPGP_signature
Description: OpenPGP digital signature