Re: Investigating a reproducibility failure

[zimoun]

Hi Konrad,

We agree on the main points in the scope of Guix. :-)  We probably
disagree on some specific points about epistemology or epistemic
justification; I am not sure to understand enough these terms to put
them here. :-)

We are far from OpenBLAS. :-)


On Wed, 16 Feb 2022 at 14:04, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Making scientific computations bit-for-bit reproducible is the moral
> equivalent of keeping a detailed lab notebook: doing your best to tell
> others exactly what you did.

A detailed lab notebook implies transparency and full control of
variability, not bit-for-bit reproducibility.

If my detailed lab notebook tracks my experiment to test gravity and
pendulum, as detailed and ideal (moral?) as it would be, i.e., providing
the capacity to build and re-build two exact same benches, then two
experiences would not provide the bit-for-bit numbers in a table
measuring the oscillations.  Because, for instance, it would depend on
the two locations, on the touch of the experimenter, etc.

In many fields, the experimental reproduction depends on the variability
of the inputs or of the instruments and therefore the scientific
community, field by field, somehow defines what “same” means, depending
on their common variability from their field.

For one, I do not see why it would be different for the computational
processing part of the experiment.  And two, asking bit-for-bit
reproducibility for one part of the experiment is asking far more than
for the others non-computational part of the same experiment.

Because I use daily computers and am deeply interested in what a
computation means, for sure, I advocate for bit-to-bit reproducibility.
But then, I discuss with my colleagues biologist or MD and somehow my
views are biased, i.e, I am trying to apply my own criteria defining
“same” from my “field” to their “field” where the same “same” must be
applied to the all chain, computational processing included.  Or at
least they have to define what is acceptable for each part.  Do not take
me wrong, such computational part must be transparent where the
variability must also be controlled, but no strictly more or totally
less than the other parts.


> When the forensics are called in, then...
>
>> Thus far, "show me the code" is the usual way to ask someone
>> what they did, and guix makes is possible to answer in great
>> detail.
>
> ... "show me the code" is not sufficient. You must also be sure that the
> code you look at is really the code that was run.

I agree.  It is “show me ALL the code” and e.g., “guix graph
python-scipy” points it is a long read. :-) Therefore, being able to
build, run, re-build and re-run are weak requirements to establish
trust.

>                                                   And that's the role of
> bit-for-bit reproducibility.

>From my understanding, the validation of a reproduction depends on
trust: what is the confidence about this or that?  Well, bit-for-bit
reproducibility is one criteria for establishing such trust.  However,
IMHO, such criteria is not the unique one, and defeating it can be
compensated by other criteria used by many experimental sciences.


Bah for what my opinion is worth on this topic. :-)

In any cases, thanks Konrad for the materials you provide about this
topic.  For the interested French reader: :-)

 - 
https://www.societe-informatique-de-france.fr/wp-content/uploads/2021/11/1024_18_2021_11.html
 - 
https://webcast.in2p3.fr/video/les-enjeux-et-defis-de-la-recherche-reproductible
 - 
https://www.fun-mooc.fr/en/courses/reproducible-research-methodological-principles-transparent-scie/

Cheers,
simon