[Octave-bug-tracker] [bug #53683] numerical output format for float values with magnitude greater than precision is wrong

[John W. Eaton]

Follow-up Comment #5, bug #53683 (project octave):

After reverting the changeset mentioned in comment #3, the behavior is


octave:1> for i = -2:8; 10^i/3, endfor
ans =  0.0033333
ans =  0.033333
ans =  0.33333
ans =  3.3333
ans =  33.333
ans =  333.33
ans =  3333.3
ans =  33333.33333
ans =  333333.33333
ans =  3333333.33333
ans =  33333333.33333


so that's not it.

What's weird about the Matlab behavior is that it sometimes shows just a few
non-zero digits, but always 4 after the decimal until it would need more than
7.  Whatever.  The "best" behavior here is pretty subjective.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53683>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/