[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Managing data files in workflows

From: Ricardo Wurmus
Subject: Re: Managing data files in workflows
Date: Mon, 03 May 2021 15:47:22 +0200
User-agent: mu4e 1.4.15; emacs 27.2

zimoun <> writes:


On Mon, 03 May 2021 at 11:18, Ricardo Wurmus <> wrote:

I’d like to introduce modifiers “immutable” and “mutable”, so that you can write “immutable file "whatever" you "want"” etc. “immutable” would take care of recording hashes and checking previously recorded hashes in a local state directory.

Ah, maybe it is same idea that we discussed some days ago on #guix-hpc.

To me, everything should be “immutable“ and stored “somewhere”.
Somehow, you do not need to hash all the contents but only hash all the
inputs hashes and the ’process’ itself.

This is in fact what we do now. Every process code snippet is hashed, and we compute the hashes of the chain of processes to determine whether to re-run something or not.

However, I miss what “mutable” could be?

It’s a very generous assumption that intermediate output files are reproducible. Often that’s not actually the case. Having a “mutable” modifier would allow us to indicate that this is in fact the case and require recomputation.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]