Re: Migrating from 3.6.4 to 3.8.0

[Philip Nienhuis]

ijourneaux wrote
> This computer has 4Gb of memory installed so there should be ~3.5Gb
> available.

If I'm not mistaken your image could be 2552 x 3296 x 3 (color channels) x
class byte size. In case of doubles (8 bytes) you need ~200 MB, in case of
uint16 it's ~50 MB. I guess that imresize() requires twice this amount (one
for the old, one for the new image). I also assume your scripts probably do
most of the computations implicitly in doubles, unless you've been very
careful and wary.
Anyway, as far as array size goes, it's nothing special it seems.

However, memory "exhaustion" problems usually come in a different way than
you seem to think. The amount of "free" RAM is a deceptive measure; it is
the degree of memory fragmentation that often bites you in the back.

Like many other programs, Octave allocates and deallocates chunks of memory
while it is running. After some time, the RAM in use occupies an alternation
of occupied and free (= freed) memory. As soon as the OS cannot find free
consecutive chunks of RAM large enough to hold your images, you get "memory
exhaustion" error messages, even though the total amount of free RAM, i.e.,
the sum of the sizes of all free chunks, still looks reassuringly
sufficient.

I often encounter similar problems with some large data structs (struct
arrays containing long time series). My strategy is to read, overwrite (in
memory) and clear the large data structs in a certain order (some
experimentation required); if that doesn't help, I restart Octave and that
has always clarified the memory issues (knocking on wood). 
This isn't Octave's fault per se; I had similar experience with 32-bit
Matlab with similar data. But with programs like Octave & Matlab is it
deceptively easy to allocate big chunks of RAM for e.g., temp arrays
(sometimes behind your back by low-level procedures) that help to fragment
memory quite fast.

So, try to reorganize your scripts in such a way that the largest data
structures are read first. Clear arrays (and big data structures) as soon as
they are no longer needed. 
In extreme cases I've had to split scripts up, save intermediate results to
one or more .mat files at the end of such a script "chunk", restart Octave
and re-read the mat files in the next script chunk.

Maybe initially preallocating a temp array a bit larger than twice your
image and clearing that just before your image is read/created/processed
might help. (I'm not sure of that as there is a difference between reserving
memory (when a variable or array is created) and actually using it
(assigning it value(s) or referencing it), and I'm not familiar with
low-level Octave internals.)

While the wait is for a 64-bit Octave version for Windows, it still seems
wise to reorganize your scripts with a critical eye for memory usage.

Philip




--
View this message in context: 
http://octave.1599824.n4.nabble.com/Migrating-from-3-6-4-to-3-8-0-tp4661903p4661924.html
Sent from the Octave - General mailing list archive at Nabble.com.