[Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow math check doesn't work correctly

[Dan Sebald]

Follow-up Comment #1, bug #54405 (project octave):

In the Array.h file is stated:


//! ### size_type
//!
//! Array::size_type is `octave_idx_type` which is a typedef for `int`
//! or `long int`, depending whether Octave was configured for 64-bit
//! indexing.


so I guess that octave_idx_type is a signed integer then.  That means what I
had printed out with std::cerr is, in fact, how the NR and NC numbers are
treated by the compiler.

I changed the overflow check in fread() to the following (note the extra
return statement added):


    // Check for overflow.
    if (nr != 0 && abs (nc) > abs (std::numeric_limits<octave_idx_type>::max
() / nr))
      {
std::cerr << "ERROR OCCURED\n";
        error ("fread: dimension too large for Octave's index type");
        return retval;
      }


and the error is caught as desired.  But still, the message is now


ERROR OCCURED
error: value on right hand side of assignment is undefined


I guess the error() message doesn't work the way I'm imagining.  Or perhaps
the large octave_idx_type values are causing strange problems elsewhere.

Nonetheless, the signed index value seems to make checking overflow a little
more complex.  I wonder if it would be better to change octave_idx_type to
unsigned and use the


    if (nr != 0 && nc > (std::numeric_limits<octave_idx_type>::max () / nr))


construct without the complicating signed math such as abs() in the location
wherever that "out of memory or dimension too large for Octave's index type"
determination is made.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54405>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/