bug#43389: 28.0.50; Emacs memory leaks

[Trevor Bentley]

I'm regularly encountering a bug that might be this one.  As with 
the previous posters, one of my emacs instances regularly grows up 
to 7-10 GB.  Garbage collection shows emacs is only aware of 
~250MB and has nothing to collect, and /proc/<pid>/smaps shows all 
of the usage in the heap.

The only emacs instance that hits this is the one I use the 
"emacs-slack" package in, which means long-lived HTTPS 
connections.  I'm aware that this is a relatively unusual use of 
emacs.

It doesn't start leaking until it has been active for 2-3 days. 
It might depends on other factors, such as suspending or losing 
network connectivity.  Once the leak triggers, it grows at a rate 
of about 1MB every few seconds. My machine has 32GB, so it gets 
pretty far before I notice and kill it. I'm not sure if there is a 
limit.

I built emacs with debug symbols and dumped some strace logs last 
time it happened.  This is from the "native-comp" branch, since 
it's the only one I had built with debug symbols:  GNU Emacs 
28.0.50, commit feed53f8b5da0e58cce412cd41a52883dba6c1be.  I see 
the same with the version installed from my package manager (Arch, 
GNU Emacs 27.1), and the strace log looks about the same, though 
without symbols.

I waited until it was actively leaking, and then ran the following 
command to print a stack trace whenever the heap is extended with 
brk():

$ sudo strace -p $PID -k -r --trace="?brk" --signal="SIGTERM"

The findings: this particular leak is triggered in libgnutls.  I 
get large batches of the following (truncated) stack trace

--- SNIP ---
> /usr/lib/libc-2.32.so(brk+0xb) [0xf6e7b] 
> /usr/lib/libc-2.32.so(__sbrk+0x84) [0xf6f54] 
> /usr/lib/libc-2.32.so(__default_morecore+0xd) [0x8d80d] 
> /usr/lib/libc-2.32.so(sysmalloc+0x372) [0x890e2] 
> /usr/lib/libc-2.32.so(_int_malloc+0xd9e) [0x8ad6e] 
> /usr/lib/libc-2.32.so(__libc_malloc+0x1c1) [0x8be51]
> /usr/lib/libgnutls.so.30.28.1(gnutls_session_ticket_send+0x566) 
> [0x3cc36] 
> /usr/lib/libgnutls.so.30.28.1(gnutls_record_check_corked+0xc0a) 
> [0x3e42a] 
> /usr/lib/libgnutls.so.30.28.1(gnutls_transport_get_int+0x11b1) 
> [0x34d31] 
> /usr/lib/libgnutls.so.30.28.1(gnutls_transport_get_int+0x3144) 
> [0x36cc4] 
> /home/trevor/applications/opt/bin/emacs-28.0.50(emacs_gnutls_read+0x5d) 
> [0x2e40a7] 
> /home/trevor/applications/opt/bin/emacs-28.0.50(read_process_output+0x28e) 
> [0x2def18] 
--- SNIP ---

A larger log file is available here: 
http://trevorbentley.com/emacs_strace.log

I'm not sure if gnutls is giving back buffers that emacs is 
supposed to free, or if the leak is entirely contained within 
gnutls, but something in that path is hanging on to a lot of 
allocations indefinitely.

Hope this is useful, and let me know if I can provide any other 
information that would be helpful.

-Trevor