bug#51787: GC takes more than 9 hours on berlin

[Christian Thäter]

On 2021-12-12 18:09, Mathieu Othacehe wrote:

>Hey,
>
>> OK, thanks. Looks it just finished removing the trash directory
>> content. I started a GC process from my session to monitor it
>> closely.  
>
>Daily GC recap:
>
>* The GC process I started yesterday, did collect 5.5TiB in
>  approximately 24 hours, that are now in the /gnu/store/trash
>  directory.
>  
>* The /gnu/store/trash directory contains 288910 entries. If those
>items
>  are removed at the same rate than on the previous days, it will take
>  days/months to delete them all.
>
>* I noticed that the upstream Nix GC process can now operate without
>  locking. I think it shouldn't be too hard to port it to our fork or
>  maybe rewrite the process in Guile while we are at it.
>
>  That will not fix the slow hard-drives issues though.

While discussing this issue on IRC I came up with some idea:

'rmrfd' a system daemon that deletes huge trees in the background where
'-rf' stands for --really --fast :)

Actually this is an use case that happens for on my backup system too.
With that idea I just started coding and ran some experiments. For me
this looks quite feasible now and I will continue next days on this
small project. Any feedback or help would be welcomed!

The initial ideas and experiments are at https://github.com/cehteh/rmrfd

Note that the important part is that it will put some efforts into
freeing as much space as possible at begin of the freeing process,
Unlike just 'rm -rf' where space may only freed really late when the
last link count of the data goes to zero.

        Cheers
                Christian