[Octave-bug-tracker] [bug #53683] numerical output format for float values with magnitude greater than precision is wrong

[Dan Sebald]

Follow-up Comment #9, bug #53683 (project octave):

This is bigger than I initially thought.  At first I was thinking a one line
fix for 4.4, but there is more to it.  Yes, there is one detail that just
doesn't make sense with regard to common definitions.  Here is fixed point
with 5 fractional digits (and is sort of what what Matlab is doing with its "4
digits after the decimal point"):


       0.00333
       0.03333
       0.33333
       3.33333
      33.33333
     333.33333
    3333.33333
   33333.33333
  333333.33333
 3333333.33333
33333333.33333


The following would be 5 significant digits, something similar to floating
point


       0.0033333
       0.033333
       0.33333
       3.3333
      33.333
     333.33
    3333.3
   33333
  333330
 3333300
33333000


Then for the sake of space, at either extreme one might use engineering
notation rather than such a long string of numerals.  

Octave is following a floating point progression at the start, then
transitions to a fixed point progression at the following point:


ans =  3333.3
ans =  33333.33333


Why should the larger number get more fractional resolution?  That's the one
point I disagree with.  Just how many significant digits there are, whether it
is fixed point or floating point is all a matter of choice.

I can see where this gets more complicated, if one considers groups of
numbers.  For example, the following illustrates that it is the smallest
number of the group that governs the overall fractional digits:


octave:7> rand(10,1) * 3e2 
ans =

   195.96153
    32.12849
   125.35201
    91.77371
     0.76179
   128.33099
    14.58701
   257.34665
   151.51730
   209.94001

octave:8> rand(10,1) * 3e2
ans =

   259.38
   281.53
   110.50
   278.94
   216.12
   116.73
   173.32
   129.39
   134.56
   288.86

octave:9> rand(10,1) * 3e2
ans =

   233.796
   131.497
   162.690
   100.161
   162.110
    25.142
   122.527
   267.176
    49.537
    79.511


Or more precisely, governed by whatever number has the most fractional digits.
 In the following there are numbers greater than that transition point
mentioned above; they therefore have five digits to the right of the decimal
point):


octave:40> rand(10,1) * 3e4
ans =

    7900.99571
    6466.43122
   18928.80875
   10963.55305
   12949.62312
   19015.19955
   22253.92189
   12301.65306
   20224.36300
   22761.48407


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53683>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/