Re: Memory leak in unix/pthread.fs

[Tomas Hlavaty]

On Tue 08 Nov 2022 at 15:38, nature <nature@blazebone.com> wrote:
> Serving multiple requests without
> blocking requires me to venture into the realm of multi-threading

another option is fork

> I prefer to create a separate thread each
> time a request comes and this led me to notice that my server wasn't
> able to serve more than ~3k requests before hanging and also the memory
> usage of my process increased a lot with each new request comming in.  I
> then proceeded to hunt down memory leaks due to the S" S\" and S+ words
> with minimal impact.  That is what lead me to realize that creating new
> threads with the PASS word was creating non-detached thread which never
> freed the memory once the thread exited.
>
> I am attaching a patch for unix/pthread.fs which fixed the big memory
> leak for me, my server is no able to server well above 3k requests
> without hanging/crashing (I tested 10 without any issues).

That is interesting.

Even with your patch, I still cannot create more than cca 500 threads.
There seem to be other resources like pipes probably hitting an ulimit.
It also seems that those do not get disposed properly, do they?

How do you dispose a thread properly in gforth?

How do you create a thread in your case?
Do you use the gforth default code which creates pipes?
Do you increase ulimit?

Would it be possible to share your example?

So far, I have only seen forth examples with fixed number of threads.

Here is my test case:

--- task3.fs

\ pthreads per request
\ (compile "gforth task3.fs")
require ~/mr/gforth/unix/pthread.fs
: seconds 1000 * ms ;
: task1 ( n -- ) dup . 10 seconds . ;
: mktask stacksize4 newtask4 ;

\ fix thread leak id:87tu39ebmv.fsf@forest.lan
\ without this, VIRT (and RES?) stays proportinal to #threads
\ with this, VIRT and RES goes down but not enough
: (pass) ( x1 .. xn n task -- ) \ gforth-experimental
    r> swap >r  save-task r@ 's !
    1+ dup cells negate  sp0 r@ 's @ -rot  sp0 r@ 's +!
    sp0 r@ 's @ swap 0 ?DO  tuck ! cell+  LOOP  drop
    pthread-id r@ 's pthread_detach_attr thread_start r> pthread_create drop ; 
compile-only
: pass ( x1 .. xn n task -- ) \ gforth-experimental
    \G activates task, and passes n parameters from the data stack
    ]] (pass) up! sp0 ! thread-init [[ ; immediate compile-only

: task& ( n -- ) 1 mktask pass task1 ;
: test ( n -- ) 0 do i task& loop ;
\ 500 test
1000 test

--- error

task3.fs:27:6: error: Invalid memory address
1000 >>>test<<<
Backtrace:
/nix/store/hj2683h26frrj7wckziclkiryacq539b-gforth-0.7.9_20220811/share/gforth/0.7.9_20220811/libcc.fs:931:14:
  0 $7FD2E1CFED00 call-c 
~/mr/gforth/unix/pthread.fs:224:18:      1 $7FD2E1D25BD0 create_pipe 
                                         2 $7FD16FF56000 
task3.fs:9:21:                           3 $7FD2E1D28930 newtask4 
task3.fs:23:20:                          4 $7FD2E1D28C48 mktask 
task3.fs:24:24:                          5 $7FD2E1D28CF8 task& 
                                         6 $1FD 
                                         7 $3E8