Re: precision of floor(a/b)

[Scott Carter]

This has nothing to do with Octave.  Neither 0.6 nor 0.1 are exactly representable in IEEE floating point, and as it turns out the ratio of their model numbers is just a hair less than 6.

So, according to IEEE arithmetic, as you might have guessed, 5 is indeed the "correct" answer to floor(0.6/0.1). 

I don't know what you are trying to do, so I don't know what the "correct" answer for what you are trying to do it (i.e., in what fashion do you know that your 0.6 is an exact value rather that 0.6 +/- some epsilon, so that the floor() function is what you want? ).  If your inputs are in decimal notation and you believe that they represent "exact" values and are not especially large, you can multiply them by a suitable power of ten *before* converting to floating point.  All the integers of reasonable value are exactly representable, and then the floor function will give you the correct value for those exact decimal fractions.

Regards,

Scott

Tim Rueth wrote:

If I do floor(6/1), I correctly get 6.  If I do floor(0.6/0.1), I get 5.  If I do floor((0.6/0.1 + 0.00000001)) I correctly get 6.

 

I assume this has to do with the precision of the division?  I read in the manual that Octave stores numbers as double precision, so this surprises me.  What's the best way to deal with this imprecision so I can always get the correct answer, or is something else going on here?

 

I'm on Windows Vista, if that makes a difference.

 

Thanks,

 

--Tim

_______________________________________________
Help-octave mailing list
address@hidden
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave
  

-- 
Scott Carter
Sr. Manager Product R&D
NKUS Lab
Nihon Kohden Corp.
Tel: 949-474-9207, X. 103
FAX: 949-474-8267
address@hidden