Re: [PATCH 1/2] Prevent excessive major gcs by having decent amount of unused heap

[Sven Hartrumpf]

Hello.

megane, 2021-01-20 17:48:
> Sven Hartrumpf <hartrumpf@gmx.net> writes:
>
>> Hello.
>>
> [snip]
>>> I need some -:hi option (only for the new GC!), otherwise it crashes as 
>>> follows:
>>>
>>>   # nallch.x32 -:a0 -:o -:s4096k 0
>>> [panic] out of memory - cannot allocate next heap segment - execution 
>>> terminated
>>
>> I have experimented some more (with x32 binaries).
>> I am now running my binary with:
>>
>> -:a0 -:hiNNNm -:hm3900m -:o -:s4096k
>>
>> Then, I tried several values NNN for -:hiNNNm.
>> crashes: 2 6 7 8 16
>> ok: 3 4 5
>>
>> I added (set-gc-report! #t) and attach a complete log from a crashing run 
>> with NNN=6.
>>
> I think you're having unlucky combination of limited memory and initial
> heap size. If I understand correctly, you should be able to get the same
> error without the patches too with some -:hi values.

That sounds reasonable.
And I managed to crash the old gc, but only for fewer GC parameter settings
(just good luck).

> You could try seeing if -:hg150 helps the situation.
>
> Btw, how much memory does the machine have?

I have a ulimit -m of 24 GB, but the machines have more RAM than that.
My problems are only for i586 and x32, so 4 GB is the theoretical maximum 
anyway.
For easier comparisons, I switched my testing from x32 to i586 completely.

> Here's a small program and a script to test how much memory you can
> allocate with different initial heap sizes:
>
> ;;; FILE: foo.scm
> (import (chicken blob))
> (define foo '())
> (let l ([tot 0] [s 1000000])
>   ;; (print "at " tot " " s)
>   (set! foo (cons (make-blob s) foo))
>   (l (+ s tot) (min 100000000 (inexact->exact (round (* s 1.4))))))
> ;;; RUN:
>
> ;;; FILE: run.sh
> #!/usr/bin/env bash
> csc5 foo.scm
> for HI in $(seq 16); do
>  echo -n "AAAA -:hi${HI}M"
>  (./foo -:g -:hi${HI}M -:hg150 2>&1) | grep -E '(at |resized)' | tail -n 1
>  echo -n "B -:hi${HI}M"
>  (./foo -:g -:hi${HI}M -:hg200 2>&1) | grep -E '(at |resized)' | tail -n 1
> done
>
> If I run this with ulimit -v 1048576 I get maximal heap sizes ranging
> from 509573502 to 1007797006 bytes w/o the patches, and
> 518025406-1024700814 with the patches.

The worst runs for your test program:
old gc AAAA -:hi1M[GC] resized heap to 102207885 bytes
new gc B   -:hi11M[GC] resized heap to  99804232 bytes
The best runs:
old gc B -:hi9M[GC] resized heap to    211858427 bytes
new gc B -:hi5M[GC] resized heap to    197589146 bytes

As hoped, the numbers for old gc and new gc are similar.
The best runs reach the optimum of 4 GB / 2.
The worst runs are somewhat disappointing, but I was not hit
by this before because I had optimized and/or lucky GC
settings for the old GC, it seems.

Thanks for your help; I know now more about wen to be cautious with
Chicken's GC.

Ciao
Sven