[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Is this a numerical problem?

From: Jordi Gutiérrez Hermoso
Subject: Re: Is this a numerical problem?
Date: Thu, 19 Aug 2010 11:36:56 -0500

2010/8/19 Judd Storrs <address@hidden>:
> 2010/8/19 Jordi Gutiérrez Hermoso <address@hidden>
>> I believe the inconsistency is how 0.999995 is displayed in each with
>> "format short". Octave displays 0.99999 and Matlab displays 1.0000,
>> from what I can read in the bug report.
> I don't understand this comparison. Matlab is displaying four digits after
> the decimal whereas octave is displaying five. If octave displayed four
> digits it would also display 1.0000.

It's five digits total. Format short displays five digtis.

> Why is Matlab more correct? The inaccuracy of the Matlab representation is
> +0.000005 and the inaccuracy of the Octave representation is -0.000005. Why
> is one "correct" and the other wrong?

Because 5 should round up. Precisely for reasons like this.

> Last time I checked GCC has very excellent binary to decimal conversion
> routines.

They're not perfect. It's a very fiddly picky thing to worry about.
Here's a blog post with links to gcc bug reports:

reply via email to

[Prev in Thread] Current Thread [Next in Thread]