[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: precision of floor(a/b)
From: |
CdeMills |
Subject: |
RE: precision of floor(a/b) |
Date: |
Tue, 6 Apr 2010 02:57:19 -0800 (PST) |
Once again, the problem is ill-defined. Knowing that 0.1 and 0.6 don't have a
finite-length representation in binary, have a look for instance in Knuth's
books , which suggest to replace the test on the real numbers
a == b
by
abs(a-b) < eps
this way, you can verify that
abs(6*.1 - .6) < eps
while
6*.1 - .6 = 1.1e-16
To avoid implementation-dependent problems, I suggest to compute your loop
as
alpha_min = 0;
alpha_max = 6;
alpha_step = 0.1;
for alpha = alpha_min : alpha_max,
# <do some stuff with alpha*alpha_step>
endfor
This way, the latest processed value will be 6*0.1, and not either .5 or .6,
depending on the way the equality test is performed. It's less elegant,
requires one more multiplication or variable, but it's predictable.
Regards
Pascal
--
View this message in context:
http://n4.nabble.com/precision-of-floor-a-b-tp1752272p1752666.html
Sent from the Octave - General mailing list archive at Nabble.com.
Re: precision of floor(a/b), Scott Carter, 2010/04/05