[Octave-bug-tracker] [bug #32924] lcm yields wrong results

[Rik]

Follow-up Comment #2, bug #32924 (project octave):

To add a bit more to Jordi's analysis, Octave is using the double class to
represent the integers for the lcm analysis.  The largest integer that can be
represented is found with the 'bitmax' function.  For standard IEEE 64-bit
doubles the result is 9.0072e+15.  The fact that you are getting results of
order 1e20 means that you have gone well beyond the ability of the double
class to represent the numbers involved.

Even using the 'uint64' class will only get you numbers up to ~1.8e19.  If you
need something more you will need to look into a package with arbitrary
precision.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?32924>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/