Re: [Bug-cpio] cpio can't read large files

[Sean Fulton]

Well, we do our backups to a 1 TB drive on the backup server, but we 
used to do the same thing to tape, so, in a nutshell:

# Now back it up to the backup host
       find $filesys -mount $SKIPSTRING -print | \
       tee $LISTFILE | /bin/cpio -o | gzip | \
       ssh -l root $BU_HOST \
       "dd of=/backups/$MYHOST/$DAY/$c_filesys.cpio.gz"

# Now send over the list for verification
       cat $LISTFILE | ssh -l root $BU_HOST \
       dd of=/backups/$MYHOST/$DAY/index.$c_filesys

# Now Verify what we sent (NOTE: we do the zcat/cpio on the localhost, 
not the BUHOST because the BUHOST handles six different machines at the 
same time. This reduces the workload while allowing us to send 
compressed data across the LAN).
       ssh -l root $BU_HOST \
       "dd if=/backups/$MYHOST/$DAY/$c_filesys.cpio.gz" | zcat | cpio 
-it >$OUTFILE

# Compare OUTFILE/LISTFILE

       diff $LISTFILE $OUTFILE >$DIFILE

If there is a $DIFILE, then that gets mailed to an admin with an alert 
message.

We've tweaked this over the years to compensate for different issues. 
The main thing we want is a list of what is getting backed up, then a 
list of what was backed up. The problem we run into by trusting the 
backup utility to make the LISTFILE is that if the backup utility 
doesn't see the file, it wouldn't show up in LISTFILE, whereas find is 
pretty thorough and gives us an independent check of what is on the "tape."

BTW: I've been reading that bzip2 might be better than gzip for 
compression because you can recover from corrupt files and it compresses 
better. Any thoughts on this?

Again, advice and help is apreciated.

sean


Sergey Poznyakoff wrote:

> Hi Sean,
>
>  
>
> > I've heard people before say to use tar for backups. My problem with
> > tar is that it did not (at least in my tests several years ago) backup
> > empty directories. We use cpio for system backups, and what makes it
> > great is that it will record *everything*.  Tar would typically skip
> > things like empty directories, creating quite a problem if you try to
> > restore an OS or web file tree.
> >    
>
> GNU tar does not skip any files (including empty directories), unless it
> is explicitely told to, so this should not create any problem. It also
> offers special mode for incremental backups and supports multi-volume
> archives. Of course, the usage of tar and cpio differ considerably, so
> it will require getting used to. If I knew how exactly your backups are
> organised, I could probably give you more specific technical advises on
> how that could be achieved with tar. 
>
> Oh, and for your information, I am going to release new versions of GNU
> cpio and GNU tar by the end of this month.
>
> Regards,
> Sergey
>
>
>  

--
Sean Fulton
GCN Publishing, Inc.
Internet Design, Development and Consulting For Today's Media Companies
http://www.gcnpublishing.com
(914) 937-2451, x203