help-octave
[Top][All Lists]

## Re: Is this a numerical problem?

 From: Lukas Reichlin Subject: Re: Is this a numerical problem? Date: Thu, 19 Aug 2010 14:32:49 +0200

```On 19.08.2010, at 13:16, Jaroslav Hajek wrote:

> On Thu, Aug 19, 2010 at 12:52 PM, Lukas Reichlin
>> Hi Jaroslav
>>
>> I realized that (in most cases, see below) floats are rounded, not truncated
>> such that -4.00004 and -3.99995 are both displayed as -4.0000. Therefore you
>> can't tell whether -4.0000 is less or greater than -4.
>>
>> octave:15> 3.9999
>> ans =  3.9999
>> octave:16> 3.99995
>> ans =  4.0000
>> octave:17> 0.99995
>> ans =  0.99995
>> octave:18> 0.999995
>> ans =  0.99999  # I'm expecting 1.00000 here
>> octave:19> 0.9999995
>> ans =  1.00000
>> octave:20> 0.999996
>> ans =  1.00000  # compare this to command 18
>>
>> Regards,
>> Lukas
>>
>
> Exactly. Which brings us back to your observation:
>
> octave:1> format long
> octave:2> a = (10 * rem (-12/5, 1))
> a = -4.00000000000000
> octave:3> a - -4.00000000000000
> ans =  8.88178419700125e-16
> octave:4> -3.99999999999999 - a
> ans =  9.32587340685131e-15
>
> which is an explanation.
>
> --
> RNDr. Jaroslav Hajek, PhD
> computing expert & GNU Octave developer
> Aeronautical Research and Test Institute (VZLU)
> Prague, Czech Republic
> url: www.highegg.matfyz.cz

Thanks for your patience. I consider the display of result 54 as an
inconsistency or a (cosmetic) bug because round (0.5) is 1. Rounding up starts
at 0.50, not at 0.51.
(Only the first digit after truncation is significant for rounding)

octave:57> 0.999996
ans =  1.00000
octave:53> 0.999995
ans =  0.99999  # this should be 1.00000 like the previous result
octave:54> 0.9999951
ans =  1.00000
octave:55> 0.9999950
ans =  0.99999  # this should be rounded up too
octave:56> 0.9999949
ans =  0.99999

Results from the "dark side":
>> 0.99995

ans =

1.0000  % this is correct

>> 0.999949

ans =

0.9999

>> 0.999950

ans =

1.0000

>> 0.999951

ans =

1.0000

>> 0.99996

ans =

1.0000

>>

Cheers,
Lukas

```