[Duplicity-talk] ISO encoded file names break list-current-files output

[Jan]

Hello

I'm using duplicity to back up a multiuser system. Some of the files I have to 
back up include characters that are not UTF-8 encoded. Typically, those 
characters have some ISO encoding. Backup and restore are working perfect, but 
I have some issues when listing the files in the current backup.

Stdout of "list-current-files" displays the files, but with question marks 
where 
the special characters should be. When using "--log-fd" or "--log-file" I got 
an exception:

-----------------%<-----------------
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.9/logging/__init__.py", line 1082, in emit
    stream.write(msg + self.terminator)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcfc' in position 
100: surrogates not allowed
Call stack:
  File "/usr/bin/duplicity", line 87, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 70, in with_tempdir
    fn()
  File "/usr/local/lib/python3.9/dist-packages/duplicity/dup_main.py", line 
1553, in main
    do_backup(action)
  File "/usr/local/lib/python3.9/dist-packages/duplicity/dup_main.py", line 
1640, in do_backup
    list_current(col_stats)
  File "/usr/local/lib/python3.9/dist-packages/duplicity/dup_main.py", line 
711, in list_current
    log.Log(user_info, log.INFO, log.InfoCode.file_list,
  File "/usr/local/lib/python3.9/dist-packages/duplicity/log.py", line 92, in 
Log
    _logger.log(DupToLoggerLevel(verb_level), s,
Message: 'Tue Aug 10 12:52:56 2021 tmp/test_ISO_\udcfcmlaut_dir'
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.9/logging/__init__.py", line 1082, in emit
    stream.write(msg + self.terminator)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcfc' in position 
107: surrogates not allowed
Call stack:
  File "/usr/bin/duplicity", line 87, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 70, in with_tempdir
    fn()
  File "/usr/local/lib/python3.9/dist-packages/duplicity/dup_main.py", line 
1553, in main
    do_backup(action)
  File "/usr/local/lib/python3.9/dist-packages/duplicity/dup_main.py", line 
1640, in do_backup
    list_current(col_stats)
  File "/usr/local/lib/python3.9/dist-packages/duplicity/dup_main.py", line 
711, in list_current
    log.Log(user_info, log.INFO, log.InfoCode.file_list,
  File "/usr/local/lib/python3.9/dist-packages/duplicity/log.py", line 92, in 
Log
    _logger.log(DupToLoggerLevel(verb_level), s,
Message: 'Tue Aug 10 12:52:56 2021 tmp/test_ISO_\udcfcmlaut_dir/subdir'
Arguments: ()
-----------------%<-----------------

Prior to duplicity 0.8 I was able to work around this issue by copying /usr/
bin/duplicity and replacing the log.Log() line with sys.stdout.write(). But 
since this code part has been moved into the module, I can't simply replace 
this anymore without running into trouble with debsums.

I tested this on Debian Bullseye with Python 3.9 and Duplicity from the Debian 
repo (0.8.17) as well as the current version (0.8.20).

How to reproduce:
1. mkdir -p "/tmp/test_ISO_$(echo -e '\xfc')mlaut_dir/subdir" (FC is an ü in 
Latin-1/ISO 8859-1)
2. backup /tmp
3. list files with "duplicity list-current-files --log-file ..."

Is there an option to solve my issue that I might have missed or does this 
have to be fixed in duplicity?

Thanks
Jan