Re: numeric error in strtonum()

[Andrew J. Schorr]

Hi,

What you are seeing is a result of the fact that gawk, by default,
stores numeric values as IEEE 754 double precision numbers.  That yields
around 16 digits of precision.  You can use the -M option to select
higher-precision calculations using GNU MPFR:

bash-5.1$ echo 0x180C66CE7576F79B | awk '{print strtonum($1)}'
1732872993550825472

bash-5.1$ echo 0x180C66CE7576F79B | awk -M '{print strtonum($1)}'
1732872993550825371

You can find more info here:
https://www.gnu.org/software/gawk/manual/html_node/Arbitrary-Precision-Arithmetic.html
and here:
https://en.wikipedia.org/wiki/Double-precision_floating-point_format

Regards,
Andy

On Fri, Nov 29, 2024 at 10:46:54AM +0100, Hans-Peter Dukek wrote:
> Dear gawk-team,
> 
> when converting nanosecond epoch timestamps (i.e. large hex numbers in 64 bit 
> entities) with the strtonum function, I discovered by coincidence that the 
> results differed from bash and dc calculations.
> 
> Version and examples can be found below.
> 
> Kind regards
> H-P
> 
> 
> hp@bullseye-server:/tmp$ uname -a
> Linux bullseye-server.local.dukek.net 5.10.0-33-amd64 #1 SMP Debian 
> 5.10.226-1 (2024-10-03) x86_64 GNU/Linux
> hp@bullseye-server:/tmp$ gawk --version
> GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)
> Copyright (C) 1989, 1991-2020 Free Software Foundation.
> 
> This program is free software; you can redistribute it and/or modify
> it under the terms of the GNU General Public License as published by
> the Free Software Foundation; either version 3 of the License, or
> (at your option) any later version.
> 
> This program is distributed in the hope that it will be useful,
> but WITHOUT ANY WARRANTY; without even the implied warranty of
> MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> GNU General Public License for more details.
> 
> You should have received a copy of the GNU General Public License
> along with this program. If not, see http://www.gnu.org/licenses/.
> hp@bullseye-server:/tmp$ echo 0x180C66CE7576F79B | awk '{print strtonum($1)}'
> 1732872993550825472
> hp@bullseye-server:/tmp$ echo $((0x180C66CE7576F79B))
> 1732872993550825371
> hp@bullseye-server:/tmp$ echo "ibase=16 ; 180C66CE7576F79B" | bc
> 1732872993550825371
> hp@bullseye-server:/tmp$ echo 0x180C66E84D8BC315 | awk '{print strtonum($1)}'
> 1732873104550249216
> hp@bullseye-server:/tmp$ echo $((0x180C66E84D8BC315))
> 1732873104550249237
> hp@bullseye-server:/tmp$ echo "ibase=16 ; 180C66E84D8BC315" | bc
> 1732873104550249237
>