help-octave
[Top][All Lists]

## Re: Is this a numerical problem?

 From: Sergei Steshenko Subject: Re: Is this a numerical problem? Date: Thu, 19 Aug 2010 04:16:09 -0700 (PDT)

```
--- On Thu, 8/19/10, Lukas Reichlin <address@hidden> wrote:

> Subject: Re: Is this a numerical problem?
> Date: Thursday, August 19, 2010, 3:52 AM
> Hi Jaroslav
>
> I realized that (in most cases, see below) floats are
> rounded, not truncated such that -4.00004 and -3.99995 are
> both displayed as -4.0000. Therefore you can't tell whether
> -4.0000 is less or greater than -4.
>
> octave:15> 3.9999
> ans =  3.9999
> octave:16> 3.99995
> ans =  4.0000
> octave:17> 0.99995
> ans =  0.99995
> octave:18> 0.999995
> ans =  0.99999  # I'm expecting 1.00000 here
> octave:19> 0.9999995
> ans =  1.00000
> octave:20> 0.999996
> ans =  1.00000  # compare this to command 18
>
> Regards,
> Lukas
>

There is a setting (I don't remember it off the top of my head) which
controls default display accuracy, i.e. default number of significant
digits.

Set it to meaningful for 'double' maximum in order to avoid such
problems.

...

The setting used to be

output_precision(14);
.

Regards,
Sergei.

```