Re: seq problem representing zero

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Simone Pascarosa on 10/8/2005 4:23 AM:
> For instance:
>> seq -3 0.6 1
> outputs:
> -3
> -2.4
> -1.8
> -1.2
> -0.6
> -1.11022e-16   <==
> 0.6
> 
...
> Maybe this is just a problem with my math libraries or similar.

No, it is just a general inherent limitation of floating point
representation, which every computer programmer should be aware of.  The
info pages also document this:

|    Be careful when using `seq' with a fractional INCREMENT, otherwise
| you may see surprising results.  Most people would expect to see `0.3'
| printed as the last number in this example:
|
|      $ seq -s ' ' 0 .1 .3
|      0 0.1 0.2
|
|    But that doesn't happen on most systems because `seq' is implemented
| using binary floating point arithmetic (via the C `double' type)--which
| means some decimal numbers like `.1' cannot be represented exactly.
| That in turn means some nonintuitive conditions like `.1 * 3 > .3' will
| end up being true.

> 
> The problem disappears using --format option; for instance:

Correct, this is one way of forcing the computer to round to less
precision before outputting, and thus cover the inaccuracies inherent in
the full precision.

- --
Life is short - so eat dessert first!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDR+aj84KuGfSFAYARApOpAJ0UYRH2urPhTC71hJj0Y5LWQh3qOgCbBMYs
JLBmwx0ffsqIp1eoMgHIXnk=
=7IBF
-----END PGP SIGNATURE-----