Re: [bug-gawk] Increased RAM memory usage in gawk 5.0.1 compared to gawk 4.1.4

[Finn Magnusson]

 Hi 
Thank you for the patch ! It solves the issue with the leaking memory !
However we have found another issue that is also contributing to high memory 
usage.It is due to a line in our code where we are doing: 
 if (mom_version !~ $1)    -> this causes RAM memory increase by over 1 GB RAM 
in gawk 5.0.1 (but not gawk 4.1.4)
when we replace this line with: if (mom_version !~ sprintf("%s",$1)   then the 
RAM usage becomes normal
The command to reproduce the fault is:gawk -f test_ram.awk -v 
mibfile=10.220.33.18_153125.log.gz -v sqlrefmapfile=sqlRefMap.txt 
And attached are the three relevant files "test_ram.awk" , 
"10.220.33.18_153125.log.gz" and "sqlRefMap.txt"
Example execution on my machine:
gawk 4.1.4: 140 MB RAM

seliius01071 [11:40] [/home/eanzmagn] -> gawk414/gawk-4.1.4/gawk -f 
test_ram.awk -v mibfile=10.220.33.18_153125.log.gz 
-vsqlrefmapfile=sqlRefMap.txt Connected to  
(SubNetwork=STP317,SubNetwork=RNC318,MeContext=RNC318,ManagedElement=1)
Last MO: 155688. Loaded 155688 MOs. Total: 155689 MOs.
  RSS COMMAND
140276 gawk414/gawk-4.1.4/gawk -f test_ram.awk -v 
mibfile=10.220.33.18_153125.log.gz -vsqlrefmapfile=sqlRefMap.txt

gawk 5.0.1: 1.27 GB RAM

seliius01071 [11:42] [/home/eanzmagn] -> gawk501/gawk-5.0.1/gawk -f 
test_ram.awk -v mibfile=10.220.33.18_153125.log.gz -vsqlrefmapfile=sqlRefMap.txt
Connected to  
(SubNetwork=STP317,SubNetwork=RNC318,MeContext=RNC318,ManagedElement=1)
Last MO: 155688. Loaded 155688 MOs. Total: 155689 MOs.
  RSS COMMAND
1270736 gawk501/gawk-5.0.1/gawk -f test_ram.awk -v 
mibfile=10.220.33.18_153125.log.gz -vsqlrefmapfile=sqlRefMap.txt

gawk 5.0.1 with change in the regexp syntax (sprintf("%s",$1) instead of $1): 
152 MB RAM
seliius01071 [11:43] [/home/eanzmagn] -> gawk501/gawk-5.0.1/gawk -f 
test_ram.awk -v mibfile=10.220.33.18_153125.log.gz 
-vsqlrefmapfile=sqlRefMap.txt -v fixram=1
Connected to  
(SubNetwork=STP317,SubNetwork=RNC318,MeContext=RNC318,ManagedElement=1)
Last MO: 155688. Loaded 155688 MOs. Total: 155689 MOs.
  RSS COMMAND
152408 gawk501/gawk-5.0.1/gawk -f test_ram.awk -v 
mibfile=10.220.33.18_153125.log.gz -vsqlrefmapfile=sqlRefMap.txt -vfixram=1

Thank you !Finn



    On Sunday, August 25, 2019, 2:09:38 PM GMT+2, <address@hidden> wrote:  
 
 Hi.

Finn Magnusson via bug-gawk <address@hidden> wrote:

> Dear gawk developers
>
> Our team recently started using gawk 5.0.1 and found that the RAM memory
> usage seems to have increased compared to gawk version 4.1.4, which we
> were using before.

Please try this fix against unmodified gawk 5.0.1, and let me know
if it also solves the problem for you. In my testing it did.

Thanks,

Arnold

---------------------------------------------------
diff --git a/builtin.c b/builtin.c
index bea88b86..15729484 100644
--- a/builtin.c
+++ b/builtin.c
@@ -735,7 +735,7 @@ format_tree(
     static const char lchbuf[] = "0123456789abcdef";
     static const char Uchbuf[] = "0123456789ABCDEF";
 
-#define INITIAL_OUT_SIZE    512
+#define INITIAL_OUT_SIZE    64
     emalloc(obuf, char *, INITIAL_OUT_SIZE, "format_tree");
     obufout = obuf;
     osiz = INITIAL_OUT_SIZE;
@@ -1647,7 +1647,8 @@ mpf1:
     }
     bchunk(s0, s1 - s0);
     olen_final = obufout - obuf;
-    if (ofre > 0)
+#define GIVE_BACK_SIZE (INITIAL_OUT_SIZE * 2)
+    if (ofre > GIVE_BACK_SIZE)
         erealloc(obuf, char *, olen_final + 1, "format_tree");
     r = make_str_node(obuf, olen_final, ALREADY_MALLOCED);
     obuf = NULL;

test_ram.awk
Description: Binary data

10.220.33.18_153125.log.gz
Description: GNU Zip compressed data

sqlRefMap.txt
Description: Text document