RE: [Groff] About numerical precision

[Ted Harding]

On 02-Apr-03 Thomas Baruchel wrote:
> I have very easely (X-x)/3+x with a good precision in scaled points.
> For instance, calling my macro with Minion-Regular 72,
>   having x = 31248
>          X = 46800
>   I get (X-x)/3+x = 36432   (ie. 36.432p)
>   which is what I expected
> 
> But, I can't manage to get \H'36432*72/46800' without any
> error: \H'\\nY*\\n(.s/\\nX' produces a multiplication overflow
> 
> and \H'36432/46800*72' seems to produce some \H'0' for obvious
> precision
> reasons.
> 
> Any idea ?

I just did

  .nr Y 36432
  .nr X 46800
  \s[72]
  .nr N \nY*\n(.s/\nX
  \nN

in groff 1.16.1 and got "56" printed out, with no overflow problems.
(Though perhaps you are using the "devdvi" device, whose device-unit
unitwidth is 131072, as opposed to 1000 for "devps" which I was using
above).

One possible approach to deal with overflow could be to use something
like

  .nr N ((\nY+5)/10)*\n(.s/((\nX+5)/10)

which will reduce the size of the integer produced by the multiplication
(and, given the sizes of integers corresponding to device units, will
not cause perceptible error in the printed output, even though the
division will drop the "remainder").

Best wishes,
Ted.


--------------------------------------------------------------------
E-Mail: (Ted Harding) <address@hidden>
Fax-to-email: +44 (0)870 167 1972
Date: 02-Apr-03                                       Time: 11:44:19
------------------------------ XFMail ------------------------------