[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Orgmode] Precision setting for spreadsheet calculation

From: Carsten Dominik
Subject: Re: [Orgmode] Precision setting for spreadsheet calculation
Date: Sat, 27 Feb 2010 00:11:31 +0100

On Feb 26, 2010, at 10:58 PM, Michael Brand wrote:

Carsten Dominik wrote:
these are the internals of calc, about which I cannot do much.


| 0.16666666666666666667 |
#+TBLFM: $1 = 1 / 2 * 3; p20 f20

shows that the internals of calc are ok, isn't it?

Hi Michael,

you are completely right, I was wrong.

The other issue that is playing into this is that when
using a printf format specifier, the number is first
converted into a Lisp number, which is a double, with
16 significant digits...  At this step, the additional
digits are lost.

Maybe you find this interesting enough so that I can
persuade you to write a FAQ entry about these issues? :-)

- Carsten

Michael Brand wrote:
But instead of

| 0.166666666666666660 |
#+TBLFM: $1 = 1 / 2 * 3; %.18f p20 f20

which I get I would expect

| 0.166666666666666667 |
#+TBLFM: $1 = 1 / 2 * 3; %.18f p20 f20

- Carsten

reply via email to

[Prev in Thread] Current Thread [Next in Thread]