bug-gnu-pspp
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

## PSPP-BUG: [bug #51235] Wrong rounding of decimals in FACTOR output?

 From: John Darrington Subject: PSPP-BUG: [bug #51235] Wrong rounding of decimals in FACTOR output? Date: Thu, 15 Jun 2017 07:33:46 -0400 (EDT) User-agent: Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0

```Follow-up Comment #3, bug #51235 (project pspp):

I did a bit of investigation into this, based on some data Jaap sent me
offline.

The number Jaap is dealing with is in fact 0.9349999999999994981791929 (at
least that's how GSL calculates it and printf ("%0.22f") displays it.)

I think the problem is in data-out.c - the function should_round_up is a bit
too naive:

/* Returns true if the magnitude represented by R should be
rounded up when chopped off at DECIMALS decimal places, false
if it should be rounded down. */
static bool
should_round_up (const struct rounder *r, int decimals)
{
int digit = r->string[r->integer_digits + decimals + 1];
assert (digit >= '0' && digit <= '9');
return digit >= '5';
}

It simply looks as the next number after the desired decimal places - which in
this case is '4' so concludes that rounding down is appropriate.

I suggest we change the implementation of should_round_up to iterate from the
right hand side and carry the previous digit.  Something like:

static bool
should_round_up (const struct rounder *r, int decimals)
{
int x;
int digit = 0;
int carry_digit = '0';
for (x = strlen (r->string) - 1;
x >= r->integer_digits + decimals + 1;
--x)
{
digit = r->string[x];
assert (digit >= '0' && digit <= '9');
if (carry_digit >= '5')
digit++;
carry_digit = digit;
}
return digit >= '5';
}

The comment for this function would need to be adjusted too, as would numerous
test results.

_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51235>

_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/

```

reply via email to

 [Prev in Thread] Current Thread [Next in Thread]