Re: [Duplicity-talk] Problem with --exclude-if-present in version 0.7.10

[Richard McGraw]

On Tue, 30 Aug 2016 11:59:35 +0100
Aaron <address@hidden> wrote:

Hello,

> Hello Richard,
> 
> On 2016-08-29 09:50, Richard via Duplicity-talk wrote:
> > I want to send a backup with --exclude-if-present=TAG and
> > --include-filelist.
> > The folder /src/folder is not accessible (mode 750, not owner, not
> > member of group). This folder is excluded by way of '- **/folder' in
> > the include filelist.  
> 
> > duplicity 0.7.10 (python 2.7.12) fails with
> > OSError: [Errno 13] Permission denied: '/src/folder/TAG'
> > 
> > duplicity 0.7.06 prints
> > Error accessing possibly locked file /src/folder  
> 
> That makes sense. 0.7.06 threw read errors if any folders within the 
> backup path were locked/unreadable, even if those were explicitly 
> excluded. In 0.7.10, it only gives the “Error accessing possibly
> locked file” if the file/folder is not excluded. You shouldn’t
> receive this error because you have excluded the folder. If you
> remove the - **/folder in the include filelist (and anything else in
> your include filelist that would exclude that folder), this error
> should come back again (please let me know if that isn’t the case).

With 0.7.10 I get the error in both cases: when '**/folder' is
excluded, and when it isn't.
Here is a stacktrace when '**/folder' is excluded in include-filelist:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1522, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 662, in incremental_backup
    bytes_written = dummy_backup(tarblock_iter)
  File "/usr/bin/duplicity", line 238, in dummy_backup
    while tarblock_iter.next():
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line
523, in next
    result = self.process(self.input_iter.next())
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line
195, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line
276, in collate2iters
    relem1 = riter1.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
179, in Iterate
    subpath, val = diryield_stack[-1].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
146, in diryield
    s = self.Select(new_path)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
208, in Select
    result = sf(path)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line
450, in exclude_sel_func
    if path.isdir() and path.append(filename).exists():
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 535,
in append
    return self.__class__(self.base, self.index + (ext,))
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 515,
in __init__
    self.setdata()
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 520,
in setdata
    self.stat = os.lstat(self.name)
OSError: [Errno 13] Permission denied: '/src/folder/TAG'


> The only difference that should be fixed, therefore, is that the
> OSError should be suppressed if the folder is separately excluded in
> a different selection function. This difference should not have any
> impact on what is backed up in this case (or any I can think of), but
> I will make a bug for this and add it to my list.
> 
> > Does the selection process resolve
> > exclude-if-present=TAG before it looks at include/exclude filelists
> > anyway ?  
> 
> As per the manual:
> --exclude-if-present filename
> Exclude directories if filename is present. Allows the user to
> specify folders that they do not wish to backup by adding a specified
> file (e.g. ".nobackup") instead of maintaining a comprehensive
> exclude/include list. This option needs to come before any other
> include or exclude options.
> 
> So you are supposed to put in that option before any other options
> (and it is therefore supposed to take priority over any other
> options). I’m not quite sure why this is mandated, as from a very
> quick look at the code it looks like it is just evaluated in order
> like the rest of the options — any ideas why this is in the manual,
> Kenneth (or anyone else)?
> 
> Kind regards,
> 
> Aaron
>