[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Octave-bug-tracker] [bug #41359] Output formatting inconsistency for in

From: Mike Miller
Subject: [Octave-bug-tracker] [bug #41359] Output formatting inconsistency for integers > 2^52
Date: Mon, 27 Jan 2014 03:15:57 +0000
User-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36

Update of bug #41359 (project octave):

                  Status:                    None => Confirmed              
                 Release:                   3.6.4 => dev                    


Follow-up Comment #1:

Yes, you are right that the values should be representable in 16 decimal
digits. The basis of the decision whether a number can be displayed as an
integer or not is failing for these cases. The output formatter then believes
that the value is not strictly an integer and displays what you are seeing.

Here are essentially the computations that the formatter is doing to determine
whether the value is an integer or not:

> a = 2^52;
> a == floor (a + .5)
ans =  1
> a-1 == floor (a-1 + .5)
ans =  1
> a+1 == floor (a+1 + .5)
ans = 0
> a+2 == floor (a+2 + .5)
ans =  1

Since we've hit the limit of the significand, adding 0.5 rolls over into the
next bit position, floor gives the wrong result, and it appears that a+1 is
not an integer according to this formula.


Reply to this item at:


  Message sent via/by Savannah

reply via email to

[Prev in Thread] Current Thread [Next in Thread]