help-octave
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Is this a numerical problem?


From: Lukas Reichlin
Subject: Re: Is this a numerical problem?
Date: Thu, 19 Aug 2010 14:32:49 +0200

On 19.08.2010, at 13:16, Jaroslav Hajek wrote:

> On Thu, Aug 19, 2010 at 12:52 PM, Lukas Reichlin
> <address@hidden> wrote:
>> Hi Jaroslav
>> 
>> I realized that (in most cases, see below) floats are rounded, not truncated 
>> such that -4.00004 and -3.99995 are both displayed as -4.0000. Therefore you 
>> can't tell whether -4.0000 is less or greater than -4.
>> 
>> octave:15> 3.9999
>> ans =  3.9999
>> octave:16> 3.99995
>> ans =  4.0000
>> octave:17> 0.99995
>> ans =  0.99995
>> octave:18> 0.999995
>> ans =  0.99999  # I'm expecting 1.00000 here
>> octave:19> 0.9999995
>> ans =  1.00000
>> octave:20> 0.999996
>> ans =  1.00000  # compare this to command 18
>> 
>> Regards,
>> Lukas
>> 
> 
> Exactly. Which brings us back to your observation:
> 
> octave:1> format long
> octave:2> a = (10 * rem (-12/5, 1))
> a = -4.00000000000000
> octave:3> a - -4.00000000000000
> ans =  8.88178419700125e-16
> octave:4> -3.99999999999999 - a
> ans =  9.32587340685131e-15
> 
> which is an explanation.
> 
> -- 
> RNDr. Jaroslav Hajek, PhD
> computing expert & GNU Octave developer
> Aeronautical Research and Test Institute (VZLU)
> Prague, Czech Republic
> url: www.highegg.matfyz.cz

Thanks for your patience. I consider the display of result 54 as an 
inconsistency or a (cosmetic) bug because round (0.5) is 1. Rounding up starts 
at 0.50, not at 0.51.
(Only the first digit after truncation is significant for rounding)

octave:57> 0.999996
ans =  1.00000
octave:53> 0.999995
ans =  0.99999  # this should be 1.00000 like the previous result
octave:54> 0.9999951
ans =  1.00000
octave:55> 0.9999950
ans =  0.99999  # this should be rounded up too
octave:56> 0.9999949
ans =  0.99999


Results from the "dark side":
>> 0.99995

ans =

    1.0000  % this is correct

>> 0.999949

ans =

    0.9999

>> 0.999950

ans =

    1.0000

>> 0.999951

ans =

    1.0000

>> 0.99996

ans =

    1.0000

>> 

Cheers,
Lukas




reply via email to

[Prev in Thread] Current Thread [Next in Thread]