Re: cross-platform backup tool Same files from different source dir causes spurious diff files

[Mr. Clif]

Hey folks,

thanks for the feedback. :-) More comments below...

On 2/7/22 8:25 PM, Robert Nichols wrote:
> On 2/7/22 7:23 PM, Leland Best wrote:
> > Hi Cliff,
> >
> > On Mon, 2022-02-07 at 11:45 -0800, Mr. Clif wrote:
> > > Hey Eric,
> > >
> > > any ideas on this? How do these diff files normally work?
> > [...]
> >
> > I'm not an 'rdiff-backup' developer or anything so all you experts 
> > out there
> > correct me if I'm wrong but ...
> >
> > IIRC 'rdiff-backup' keeps inode info as part of the metadata for each 
> > file.
> > When you mount a filesystem Linux assigns "fake" inode numbers to avoid
> > collisions between filesystems on different devices/partitions/etc..  
> > So if you
> > change the mount point, every file could potentially get a new inode 
> > number and,
> > consequently, have changed metadata.  That results in 'rdiff-backup' 
> > creating a
> > '*.diff*' file for every source file.
>
> Device and inode metadata is kept only for files with multiple hard 
> links. That's
> to keep track of which links reference the same file. That information 
> is not
> needed for files with just a single hard link, and unless something 
> has changed
> in the latest release that metadata is not kept. You can look in the
> mirror_metadata file (it's compressed ASCII) and see what fields are 
> present
> for each file.

Cool, these are the diff.gz files? I tried ungzipping them but the first 
"line" of data still seems to be binary. Is it encoded somehow?

> > In addition, since 'rdiff-backup' now thinks the files may have 
> > changed it
> > spends a lot of time checking if anything other than metadata has 
> > changed which
> > _might_ account for the apparently low throughput.
>
> That would definitely be true, and the presence of all those "zero 
> diff" files
> show that it is, in fact, happening.

Ok lets see if I understand, inode data is usually not stored, but 
because it's on a different mount point it thinks "something" has 
changed so it thoroughly checks everything. I'm wondering though 
afterwards, if it could compare the current state to the previous one, 
and not create the new mirror_metadata for the files that haven't 
actually changed. Or... is there something I'm missing? Maybe they serve 
a purpose, or the way it's written makes it's hard to get rid of them. 
Just curious. :-)

Yes Eric Lavarde, it probably is a one time effect. However... I don't 
think a readonly filesystem could be confused with file metadata 
changes, and I think that the last access time of a file is probably 
ignored in the metadata comparison.

By the way, I was going to say I didn't have a problem with slow 
throughput, that was Eric Robinson's thread. Though now that I think 
about it, it took quite awhile to thoroughly check the whole filesystem 
if that's what it was doing, and of course there's the additional 500M. ;-)

    Thanks for a great project,
    Clif