Re: cross-platform backup tool Same files from different source dir causes spurious diff files

[Mr. Clif]

Hi Folks,

In case it's helpful, here's a shell function I came up with to backup 
snapshots of VMs in a way that preserves the usual UIDs and GIDs:

#
# Backup a vm
# Called with the hostname, logical volume that the vm uses, and the 
rdiff archive to put the changes in.
#

backup_vm() {
ssh $1 /root/backup_pre.pl     # Handle anything that needs to be done 
first on the remote end.

# Create a snapshot for this VM.
/sbin/lvcreate -s -L3000 -n ${2}-snapbaklv /dev/FooVG/${2}
mount -o ro /dev/FooVG/${2}-snapbaklv /mnt/vm-backup-snapshot

# Shift the UIDs and GIDs in the namespace back down to normal
bindfs -r --multithreaded --uid-offset=-100000 --gid-offset=-100000 
/mnt/vm-backup-snapshot/ /mnt/vm-backup-bindfs/

# The old way of backing things up
# /usr/bin/rdiff-backup --preserve-numerical-ids --exclude-sockets 
--exclude-other-filesystems \
#  --include-globbing-filelist include-list ${1}::/ /mnt/${3}

# Do the backup
/usr/bin/rdiff-backup --preserve-numerical-ids --exclude-sockets 
--exclude-other-filesystems \
  --user-mapping-file mapping_file --group-mapping-file mapping_file 
--include-globbing-filelist include-list \
  /mnt/vm-backup-bindfs/ /mnt/${3}

# Cleanup!
umount /mnt/vm-backup-bindfs; umount /mnt/vm-backup-snapshot
/sbin/lvremove -f /dev/FooVG/${2}-snapbaklv
}

And you'd call it like so:

backup_vm myserver vm-109-disk-0 myserver-rdiff

Of course you probably would have to make some adjustments for your 
situation, and perhaps you have something better to use than bindfs. Oh 
and some will notice that I'm trying to get away with using both 
--preserve-numerical-ids and --user-mapping-file (for exceptions), so 
far it seems to be working. In my case there are only two or three files 
that might benefit from being handled as exceptions.

    Ciao,
    Clif

On 2/9/22 11:19 PM, Mr. Clif wrote:
> Interesting,
>
> If it helps I would say my use case is trying to preserve the 
> "correct" view of the metadata, not the shifted view presented through 
> a locally mounted logical volume. The backups I'm hoping to create 
> whether from the running vm or a snapshot can always be used to 
> restore parts back into the running vm. Right now my backups for this 
> vm have been corrupted by the shifted UID/GIDs I can no longer use 
> that archive to restore to the running vm.
>
> Sure if I wanted to stop the vm then mount it's system image I could 
> restore to that, but I'd rather not have to reboot my system just to 
> get some files back.
>
> I guess I'm also suggesting that the --user-mapping-file is 
> bidirectional and undoes those changes during a restore.
>
>     Best,
>     Clif
>
> On 2/9/22 10:30 PM, ewl+rdiffbackup@lavar.de wrote:
> > Hi,
> >
> > On 09/02/2022 21:46, Mr. Clif wrote:
> > > It would be great if rdiff-backup would allow offsets like this or 
> > > even better the ability to specify a range like
> > > 100000-165535:0-65535
> > > Or you could just have the starting UID after the colon.
> > >
> > > In the man page under USERS AND GROUPS, it says:
> > > "If you specify both --preserve-numerical-ids and one of the mapping 
> > > options, the behavior is undefined."
> > >
> > > I think it would be better to allow both with the user-mapping-file 
> > > overriding the preserve-numerical-ids behavior when necessary. As in 
> > > my use case I never want user name mapping.
> >
> > The user mapping file is only used to save the files on disk, it 
> > isn't used in the metadata so it wouldn't help in your case.
> > Anyway, we already have an enhancement record in this regard: 
> > https://github.com/rdiff-backup/rdiff-backup/issues/621
> >
> > I'm personally not sure that it's a good idea: a backup is there to 
> > save things alike, not to modify them, so it's a dangerous road IMHO.
> >
> > KR, Eric