[rdiff-backup-users] [Errno 28] No space left on device - but which device?

[Ron Leach]

List, good afternoon, we're puzzled by this error (from our overnight 
backup) because the backup destination has more than 350 GB free, and 
the increment would likely have been somewhere between 50MB and 200MB. 
 It was a 'fatal' error, and the backup ceased there and then.

The nightly backup is scheduled by cron and has been completed for a 
couple of years without a fatal error.  Here's the command:

rdiff-backup --print-statistics -v3 --exclude /Data/.Trash-1000 /Data 
/mnt/D7bkp100/Backups

/Data is a local 1.8TB partition, and has around 163GB spare.

/mnt/D7bkp100/ is an NFS export of a 2.6TB LVM located on another 
machine, and has 367GB spare.

Here's a df of the machine doing the backup:

address@hidden;13:50:09;~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              2.8G  2.8G   24K 100% /
tmpfs                 501M     0  501M   0% /lib/init/rw
udev                   10M  792K  9.3M   8% /dev
tmpfs                 501M     0  501M   0% /dev/shm
/dev/md6              1.8T  1.7T  163G  92% /Data
/dev/sda1             324M   12M  296M   4% /boot
/dev/sdb1             313M   16M  281M   6% /boot2
/dev/md5               38G  2.9G   35G   8% /home
/dev/md4              949M  4.3M  945M   1% /tmp
/dev/md2              9.4G  1.2G  8.2G  13% /usr
/dev/md3              4.7G  1.1G  3.7G  23% /var
192.168.0.100:/mnt/R1/bkp100
                      2.6T  2.1T  367G  86% /mnt/D7bkp100
address@hidden;13:50:13;~$

To check that there really was (and is) enough space on the 
destination machine, I copied a 4.4GB Debian iso from /Data to 
/mnt/D7bkp100, and it copied without problems.

I'm a little puzzled as to why rdiff-backup encountered a problem, and 
would be grateful for any suggestions as to where I might look for 
some more clues.

In that spirit, I include [1] the log output (it's not too long), but 
it seems to only list the hierarchy of procedure calls.  (I've never 
provided logs before - if there is a more convenient way to parse or 
format it let me know and I'll repost it.)

If anyone can determine from it what 'mode' rdiff-backup was in when 
it hit the problem, that might help provide a clue as to which 
resource was encountered as full.

Grateful for any comments,

regards, Ron

[1]
Exception '[Errno 28] No space left on device' raised of class '<type 
'exceptions.IOError'>':
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
304, in error_check_Main
    try: Main(arglist)
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
324, in Main
    take_action(rps)
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
346, in Backup
    backup.Mirror(rpin, rpout)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 38, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 232, in patch
    ITR(diff.index, diff)
  File "/var/lib/python-support/python2.5/rdiff_backup/rorpiter.py", 
line 281, in __call__
    last_branch.fast_process(*args)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 529, in fast_process
    if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 553, in patch_to_temp
    result = self.patch_snapshot_to_temp(diff_rorp, new)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 582, in patch_snapshot_to_temp
    (diff_rorp, new))
  File "/var/lib/python-support/python2.5/rdiff_backup/robust.py", 
line 32, in check_common_error
    try: return function(*args)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 105, in copy
    if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 133, in copy_reg_file
    return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 1195, in write_from_fileobj
    copyfileobj(fp, outfp)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 64, in copyfileobj
    outputfp.write(inbuf)

IOError while trying to log exception!
Exception '[Errno 28] No space left on device' raised of class '<type 
'exceptions.IOError'>':
  File "/var/lib/python-support/python2.5/rdiff_backup/robust.py", 
line 32, in check_common_error
    try: return function(*args)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 105, in copy
    if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 133, in copy_reg_file
    return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 1195, in write_from_fileobj
    copyfileobj(fp, outfp)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 64, in copyfileobj
    outputfp.write(inbuf)

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
304, in error_check_Main
    try: Main(arglist)
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
324, in Main
    take_action(rps)
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/var/lib/python-support/python2.5/rdiff_backup/Main.py", line 
346, in Backup
    backup.Mirror(rpin, rpout)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 38, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 232, in patch
    ITR(diff.index, diff)
  File "/var/lib/python-support/python2.5/rdiff_backup/rorpiter.py", 
line 281, in __call__
    last_branch.fast_process(*args)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 529, in fast_process
    if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 553, in patch_to_temp
    result = self.patch_snapshot_to_temp(diff_rorp, new)
  File "/var/lib/python-support/python2.5/rdiff_backup/backup.py", 
line 582, in patch_snapshot_to_temp
    (diff_rorp, new))
  File "/var/lib/python-support/python2.5/rdiff_backup/robust.py", 
line 32, in check_common_error
    try: return function(*args)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 105, in copy
    if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 133, in copy_reg_file
    return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 1195, in write_from_fileobj
    copyfileobj(fp, outfp)
  File "/var/lib/python-support/python2.5/rdiff_backup/rpath.py", 
line 64, in copyfileobj
    outputfp.write(inbuf)
IOError: [Errno 28] No space left on device
close failed: [Errno 28] No space left on device