Re: [lmi] rate_table_tool: merge a whole directory

[Greg Chicares]

On 2016-11-23 01:29, Greg Chicares wrote:
> Proprietary rate tables are distributed along with our free software.
> These tables can be stored in two interconvertible formats:
>  - binary: not human-readable, which is good for distribution to
>    outsiders who should be discouraged from extracting proprietary
>    data, but inconvenient for internal maintenance; and
>  - human-readable text, which is better for maintenance but not for
>    external distribution.
> Our plan is to store tables as text in a proprietary git database,
> and use 'rate_table_tool' to convert them to binary for distribution.

Testing this, I see some anomalies. Let
  {old, new} be rate databases, and
  {txt, bin} be text and binary formats.
I convert
  old_bin --> old_txt --> new_bin
and then run a system test using new_bin in place of old_bin, and
observe several regression errors.

Nevertheless, if I convert
  old_bin --> old_txt --> new_bin
as above, then
  new_bin --> new_txt
then all old_txt and new_txt files are identical. I can't see any
anomaly there, but clearly old_bin and new_bin are not identical,
so either (1) conversion to text involves rounding that removes
any discrepancies, or (2) the code in 'actuarial_table.cpp' behaves
differently than that in 'rate_table.cpp'.

Digging into a particular, simple example [this is proprietary
test case 'XXXXc2005.000000009.test' where "XXXX" = redaction]:

  issue age 39
  SevenPayTable 262
  table '262.txt' contains the line:
    39  0.04551
  specified amount 70000000
  expected: 70000000 * 0.04551 = 3185700.00 exactly
  observed: 3185700.00 with old_bin [exactly as expected]
  observed: 3185699.99000000022352 with new_bin [anomaly]

lmi rounds the calculated seven-pay premium down to cents, so a
difference of 1 ulp in the binary number read from the table could
account for this anomaly. I imagine a 1 ulp difference could arise
in table_impl::parse_single_value() [showing only the most relevant
lines]:

    auto const res_int_part = strict_parse_number(current);
    auto const res_frac_part = strict_parse_number(res_int_part.end + 1);

    double value = res_frac_part.num;
    value /= std::pow(10, *num_decimals_);
    value += res_int_part.num;
    return value;

My hypothesis then, comparing floating-point numbers for strict
equality, is that:
  assert(0.04551 != 4551 / 10000)

I suspect it's necessary to parse text as above in order to write
helpful diagnostics, but I think the return value should be the
applicable character substring converted by some other means, e.g.,
  value_cast<double>(std::string const&)
I can figure out what conversion we need. Vadim, can you show me
how to get a std::string object containing the formatted single
number that is to be converted (because I'm snow-blind after
staring at all these pointers)?

(The early exit that I added in case the datum really is integral
seems safe: converting strings to integers is exact.)