help-octave
[Top][All Lists]

## Re: Is this a numerical problem?

 From: Lukas Reichlin Subject: Re: Is this a numerical problem? Date: Thu, 19 Aug 2010 12:52:46 +0200

```Hi Jaroslav

I realized that (in most cases, see below) floats are rounded, not truncated
such that -4.00004 and -3.99995 are both displayed as -4.0000. Therefore you
can't tell whether -4.0000 is less or greater than -4.

octave:15> 3.9999
ans =  3.9999
octave:16> 3.99995
ans =  4.0000
octave:17> 0.99995
ans =  0.99995
octave:18> 0.999995
ans =  0.99999  # I'm expecting 1.00000 here
octave:19> 0.9999995
ans =  1.00000
octave:20> 0.999996
ans =  1.00000  # compare this to command 18

Regards,
Lukas

On 19.08.2010, at 11:43, Jaroslav Hajek wrote:

> On Thu, Aug 19, 2010 at 11:31 AM, Lukas Reichlin
>> Hi Jaroslav
>>
>> Thanks for your quick reply. # a < -4 is meant as a comment about what I've
>> learned at elementary school ;-) -4.0000...001 is smaller than -4 while
>> -3.99999… is greater than -4.
>
> Okay, you've been taught well :) So, if a number is displayed as
> -4.00000000000000, does it in reality look like -4.0000...001 or
> -3.99999...?
>
> --
> RNDr. Jaroslav Hajek, PhD
> computing expert & GNU Octave developer
> Aeronautical Research and Test Institute (VZLU)
> Prague, Czech Republic
> url: www.highegg.matfyz.cz

```