[Octave-bug-tracker] [bug #36133] num2str displays more than 16 significant digits for large integer inputs

[John W. Eaton]

Follow-up Comment #9, bug #36133 (project octave):

15-17 decimal digits is true for arbitrary numbers, but powers of 2 can be
converted to decimal exactly, up to 2^1023.  So limiting to 15-17 decimal
digits doesn't make sense for ALL 64-bit IEEE floating point values.

But yes, we should be switching to exponential display using some kind of
rules that produce "pleasing" results.  One problem is deciding what those
rules should be.

What does Matlab do for num2str (2^512)?  What about sprintf ('%.0f', 2^512)?

Also, if we "clean up" the output form our printf functions, then  they will
differ from what people will get when they use C++ streams or C stdio
functions.  Is that really what we want?  If so, then we might want to make
our functions easily available for use in .oct files.

A related issue is the max precision problem in bug #53456.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?36133>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/