[Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow math check doesn't work correctly

[Dan Sebald]

Follow-up Comment #19, bug #54405 (project octave):

@JWE, just a thought.  Here's what gcc compiler does when trying to take the
absolute value of minimum int64 value:


  CXX      libinterp/corefcn/libinterp_corefcn_libcorefcn_la-oct-stream.lo
/home/sebald/octave/octave/octave/libinterp/corefcn/oct-stream.cc:6610:55:
warning: integer constant is so large that it is unsigned
 std::cerr << "labs(-9223372036854775808) = " << labs(-9223372036854775808) <<
"


Somehow it's figuring out from the ASCII string that it won't work.  Octave
could do a similar check on a manually entered integer index value rather than
convert it to float.  That is, say the user types:

zeros(92233720368547758083333)

The error could be that the index is too large, rather than out of memory. 
Huh, I just tried this and


octave:1> zeros(92233720368547758083333)
ans = [](0x0)
octave:2> zeros(9223372036854775808333)
ans = [](0x0)
octave:3> zeros(922337203685477580833)
ans = [](0x0)
octave:4> zeros(92233720368547758083)
ans = [](0x0)
octave:5> zeros(9223372036854775808)
ans = [](0x0)
octave:6> zeros(922337203685477580)
error: out of memory or dimension too large for Octave's index type
octave:6> zeros(922337203685477587)
error: out of memory or dimension too large for Octave's index type
octave:6> zeros(922337203685477588)
error: out of memory or dimension too large for Octave's index type
octave:6> 


Possible bug?

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54405>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/