[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Why does this happen
From: 
John W. Eaton 
Subject: 
Why does this happen 
Date: 
Tue, 13 Oct 1998 10:35:36 0500 (CDT) 
On 13Oct1998, Preben Randhol <address@hidden> wrote:
 I have octave 2.0.13 on a Linux 2.0.32 with RedHat 5.0 (glibc)

 I cannot understand why this happens:

 octave:2> B=[7973.6;.96500;2.7140]
 B =

 7973.60000
 0.96500
 2.71400

 octave:3> format long
 octave:5> B
 B =

 7973.600000000000364
 0.965000000000000
 2.714000000000000

 Why is B(1) now suddenly 7973.600000000000364 and not
 7973.600000000000000 ?
The short answer is that the binary floating point format that your
computer uses to store numbers can only represent some decimal values
exactly. In your example, you've asked for more digits than you have
bits to represent them 7973.6, and the formatting code in the C/C++
library is doing the best job it can given limited precision of the
binary floating point data. Note that the other numbers are also not
represented exactly. If you ask for more digits, you will see similar
results for all the numbers in your B vector:
Octave, version 2.0.13 (i686pclinuxgnulibc1).
Copyright (C) 1996, 1997, 1998 John W. Eaton.
This is free software with ABSOLUTELY NO WARRANTY.
For details, type `warranty'.
octave:1> output_max_field_width = 40;
octave:2> output_precision = 30;
octave:3> B=[7973.6;.96500;2.7140]
B =
7973.600000000000363797880709171295
0.964999999999999968913755310496
2.713999999999999968025576890795
Here is an example of a decimal fraction that can be represented
exactly:
octave:4> 1.25
ans = 1.25000000000000000000000000000
You might argue that the binary to decimal conversion code should do
something different, but first you should probably read the code in
floatconv.c in the Linux C/C++ I/O library along with the paper that
is referenced there:
Inspired loosely by William D. Clinger's paper "How to Read Floating
Point Numbers Accurately" [Proc. ACM SIGPLAN '90, pp. 92101].
Welcome to the wonderful world of floating point arithmetic. :)
jwe