Re: Investigating a reproducibility failure

[Konrad Hinsen]

Hi Simon,

> We are far from OpenBLAS. :-)

That's fine with me. The more distance between me and OpenBLAS, the
happier I am ;-)

> On Wed, 16 Feb 2022 at 14:04, Konrad Hinsen <konrad.hinsen@fastmail.net> 
> wrote:
>
>> Making scientific computations bit-for-bit reproducible is the moral
>> equivalent of keeping a detailed lab notebook: doing your best to tell
>> others exactly what you did.
>
> A detailed lab notebook implies transparency and full control of
> variability, not bit-for-bit reproducibility.

That's why I said "moral" equivalent. Computations are different from
experiments. Typical mistakes are different, and technical possibilities
are different.

1. You can't have the equivalent of bit-for-bit reproducibility with
   experiment. You can with computers, and with good tool support
   (Guix!)  it can become a routine task that takes little
   effort. So... why *not* do it?

2. A computation involves many more details than any typical experiment.
   Just writing down what you did is *not* enough for documenting a
   computation, as experience has shown. So you need more than the
   lab notebook. If your computation is bit-for-bit reproducible, you
   know that you have documented every last detail. Inversely, if you
   cannot reproduce to the bit level, you know that *something* is out
   of your control.

In the end, my argument is more pragmatic than philosophical. If
bit-for-bit reproducibility is (1) useful for resolving issues in the
future, and (2) cheap to get with good tool support, then we should
go for it.

The main reason why people argue against it is lack of tool support in
their work environments. They conclude that it's a difficult goal to
achieve, and then start to reason that it's not strictly necessary for
the scientific method. Which is true. But... it's still very useful.

>> And that's the role of bit-for-bit reproducibility.
>
> From my understanding, the validation of a reproduction depends on
> trust: what is the confidence about this or that?  Well, bit-for-bit
> reproducibility is one criteria for establishing such trust.  However,
> IMHO, such criteria is not the unique one, and defeating it can be
> compensated by other criteria used by many experimental sciences.

Definitely. But in many cases, bit-for-bit reproducibility is the
cheapest way to build trust, given good tool support. In other cases,
e.g. HPC or exotic hardware, it's expensive, and then you look for
something else.

Cheers,
  Konrad.