[Octave-bug-tracker] [bug #63803] Saving causes OOM, crash, and loss of already saved data

[Arun Giridhar]

Follow-up Comment #4, bug #63803 (project octave):

I instrumented the octave_bool_matrix::save_binary function in the
ov-bool-mat.cc file as shown in the attached hg diff. If you use it, add this
to your configure line when you build Octave: LIBS="-lprocps".

The big result is that it *triples* the memory usage, not just double. So when
I was trying to write about 7.5 GB in comment #0, it was using roughly 23 GB
of memory, which was a little beyond my RAM + swap capacity after taking into
account the memory already used to run OS services, hence an OOM crash in the
middle of the save, corrupting the saved file.


octave:1> arr = false (1e5, 5e4);

octave:2> whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  ===== 
         arr    100000x50000             5000000000  logical

Total is 5000000000 elements using 5000000000 bytes

octave:3> nnz (arr)
ans = 0

octave:4> save -binary test2.mat
../libinterp/octave-value/ov-bool-mat.cc:369 Usage: 5571063808
../libinterp/octave-value/ov-bool-mat.cc:374 Usage: 5571063808
../libinterp/octave-value/ov-bool-mat.cc:379 Usage: 10571067392
../libinterp/octave-value/ov-bool-mat.cc:384 Usage: 10571067392
../libinterp/octave-value/ov-bool-mat.cc:389 Usage: 15571070976
../libinterp/octave-value/ov-bool-mat.cc:395 Usage: 15571070976
../libinterp/octave-value/ov-bool-mat.cc:400 Usage: 15571070976

octave:5> whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  ===== 
         ans         1x1                          8  double
         arr    100000x50000             5000000000  logical

Total is 5000000001 elements using 5000000008 bytes

octave:6> exit


To decode the line numbers above, one copy is created by this line:

bool *mtmp = m.fortran_vec ();

and another by this line:

OCTAVE_LOCAL_BUFFER (char, htmp, nel);


To build on nrjank's points in comment #1:

* For robustness, can we write to a temporary and then rename it to the given
filename, as opposed to writing to that file directly? This might consume more
disk space during the save, but that's better and more robust than tripling
the RAM usage during the save and crashing.

* There are some 44 or so save_binary functions for all the different data
types. Should each of them be fixed individually or is there a way to fix
multiple types at once to avoid the creation of temporaries?

* I cannot figure out why it creates *two* temporaries. Shouldn't mtmp be just
a pointer to the fortran_vec, not yet another copy of it? Why does calling
fortran_vec() trigger a copy to be created?


(file #54357)

    _______________________________________________________

Additional Item Attachment:

File name: memory.patch                   Size:1 KB
    <https://file.savannah.gnu.org/file/memory.patch?file_id=54357>



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63803>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/