[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Octave-bug-tracker] [bug #49699] Octave crashes when using fread to loa
From: |
anonymous |
Subject: |
[Octave-bug-tracker] [bug #49699] Octave crashes when using fread to load more than 2 GB from a file |
Date: |
Fri, 25 Nov 2016 13:17:12 +0000 (UTC) |
User-agent: |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36 |
Follow-up Comment #2, bug #49699 (project octave):
Thanks, Rik, that explains what to do to fix the problem (though Octave should
just produce an error, not crash).
By the way, that wiki article is very misleading. The limitation is *not* 2 GB
of memory, but rather 2^31 *elements*.
In an installation not compiled with <code>--enable-64</code> this produces an
error as expected:
<pre>
>> a = zeros (1024*1024*2048, 1, 'int8');
error: out of memory or dimension too large for Octave's index type
</pre>
If we reduce the volume by just one slice, we do not hit the limit. But if we
also change the data type a larger one the memory used by the array can be
much larger:
<pre>
>> a = zeros (1024*1024*2047, 1, 'int8');
>> b = zeros (1024*1024*2047, 1, 'double');
>> whos
Variables in the current scope:
Attr Name Size Bytes Class
==== ==== ==== ===== =====
a 2146435072x1 2146435072 int8
b 2146435072x1 17171480576 double
Total is 4292870144 elements using 19317915648 bytes
</pre>
Both arrays have a slightly less than 2^31 elements. But while array
<code>a</code> is just under 2 GB, <code>b</code> is close to 16 GB.
That is the reason why I'm able to workaround the limitation by reading the
file in batches of less than 2 GB: my files and arrays are larger than 2 GB,
but the number of elements is less than 2^31 because I use float32 and int16.
_______________________________________________________
Reply to this item at:
<http://savannah.gnu.org/bugs/?49699>
_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/