help-octave
[Top][All Lists]

## Why does this happen

 From: John W. Eaton Subject: Why does this happen Date: Tue, 13 Oct 1998 10:35:36 -0500 (CDT)

```On 13-Oct-1998, Preben Randhol <address@hidden> wrote:

| I have octave 2.0.13 on a Linux 2.0.32 with RedHat 5.0 (glibc)
|
| I cannot understand why this happens:
|
| octave:2> B=[7973.6;.96500;2.7140]
| B =
|
|   7973.60000
|      0.96500
|      2.71400
|
| octave:3> format long
| octave:5> B
| B =
|
|   7973.600000000000364
|      0.965000000000000
|      2.714000000000000
|
| Why is B(1) now suddenly 7973.600000000000364 and not
| 7973.600000000000000 ?

The short answer is that the binary floating point format that your
computer uses to store numbers can only represent some decimal values
exactly.  In your example, you've asked for more digits than you have
bits to represent them 7973.6, and the formatting code in the C/C++
library is doing the best job it can given limited precision of the
binary floating point data.  Note that the other numbers are also not
represented exactly.  If you ask for more digits, you will see similar
results for all the numbers in your B vector:

Octave, version 2.0.13 (i686-pc-linux-gnulibc1).
Copyright (C) 1996, 1997, 1998 John W. Eaton.
This is free software with ABSOLUTELY NO WARRANTY.
For details, type `warranty'.

octave:1> output_max_field_width = 40;
octave:2> output_precision = 30;
octave:3> B=[7973.6;.96500;2.7140]
B =

7973.600000000000363797880709171295
0.964999999999999968913755310496
2.713999999999999968025576890795

Here is an example of a decimal fraction that can be represented
exactly:

octave:4> 1.25
ans = 1.25000000000000000000000000000

You might argue that the binary to decimal conversion code should do
something different, but first you should probably read the code in
floatconv.c in the Linux C/C++ I/O library along with the paper that
is referenced there:

Inspired loosely by William D. Clinger's paper "How to Read Floating
Point Numbers Accurately" [Proc. ACM SIGPLAN '90, pp. 92-101].

Welcome to the wonderful world of floating point arithmetic.  :-)

jwe

```