Re: Sieve, problem backing up mail

[Sergey Poznyakoff]

Hi Steve,

Thanks for the additional info.

It turned out the issue is due to different ways of encoding message
UIDs implemented by mailutils and mbsync.  Mbsync uses message UIDs
to track which messages have been already downloaded and which are
new.  The problem is described in mbsync manpage, section "Maildir
Stores":

  As mbsync needs UIDs, but no standardized UID storage scheme exists for
  Maildir, mbsync supports two schemes, each with its pros and cons.
  The native scheme is stolen from the latest Maildir patches to c-client
  and  is therefore compatible with pine. The UID validity is stored in
  a file named .uidvalidity; the UIDs are encoded in the file names of the
  messages.

(sorry for a lengthy quote).  Now, the way mbsync encodes UIDs is by
storing a ",U=<UID>" attribute in the file name.  This is very similar
to the technique used by mailutils, except that it uses lowercase "u":
",u=<UID>".  As a consequence, message UIDs as perceived by mbsync and 
mailutils differ.  The story is further complicated by the fact that
mailutils implements the so-called "modern delivery identifiers", while
mbsync uses "old-fashioned" ones (both terms are from the original
description of the maildir format by D. J. Bernstein [1]).

To cope with this, I have pushed a change that enables mailutils to
detect mbsync-style mailboxes and to modify them in a way understandable
to mbsync.  Now mailutils automatically determines the "compatibility
flavor" of the mailbox it is opening.  Once determined, it proceeds
operating with the mailbox in a way compatible with the software that
created it.  The change is available at [1].  Notice, that the main
GNU git server is now under a constant load from ai bots, so its response
times leave much to be desired.  You'd be better off using its mirror
instead [2].

That fixes the main problem reported in your original letter.  There
remains, however, a problem with the general approach.  Let me
illustrate it.

Running your recipe with modified mailutils (modulo file path and
account changes) I get: 

  # First sync:
  $ mbsync --config mbsyncrc myaccount-inbox
  Processed 1 box(es) in 1 channel(s),
  pulled 100 new message(s) and 0 flag update(s),
  expunged 0 message(s) from near side,
  pushed 0 new message(s) and 0 flag update(s),
  expunged 0 message(s) from far side.

  # Sieving mailbox:
  $ sieve -v --mbox-url=./mail/INBOX ./filter.siv
  sieve: ./filter.siv:7.3-37: FILEINTO on msg uid 1: delivering into 
maildir:///.../account/backup-allmail
  sieve: ./filter.siv:8.3-6: KEEP on msg uid 1
  ...
  sieve: ./filter.siv:7.3-37: FILEINTO on msg uid 100: delivering into 
maildir://.../account/backup-allmail
  sieve: ./filter.siv:8.3-6: KEEP on msg uid 100

So far so good.  Now, after sending a new message to my remote account:

  # Second sync:
  $ mbsync --config mbsyncrc myaccount-inbox
  Processed 1 box(es) in 1 channel(s),
  pulled 1 new message(s) and 1 flag update(s),
  expunged 1 message(s) from near side,
  pushed 0 new message(s) and 0 flag update(s),
  expunged 0 message(s) from far side.

mbsync downloads only the new message, which is what we aimed for.

Then, calling sieve second time, I got:

  $ sieve -v --mbox-url=./mail/INBOX ./filter.siv
  sieve: ./filter.siv:7.3-37: FILEINTO on msg uid 1: delivering into 
maildir:///.../account/backup-allmail
  sieve: ./filter.siv:8.3-6: KEEP on msg uid 1
  ...
  sieve: ./filter.siv:7.3-37: FILEINTO on msg uid 101: delivering into 
maildir://.../account/backup-allmail
  sieve: ./filter.siv:8.3-6: KEEP on msg uid 101

And here's the rub: since all processed messages are kept in the
original mailbox, sieve happily processes them again and appends all of
them to backup-allmail second time.  In contrast to mbsync, sieve is not
a mail synchronization tool, and it does not use uids to track which
mails have been processed and which haven't.  The solution would be
either to change the backup approach, or to implement an extension to sieve
that would make it uid-aware.  The latter seems to be promising.  I'll
investigate it when I have enough free time.  Feel free to ping me, if
it takes too long.

Regards,
Sergey

[1] http://cr.yp.to/proto/maildir.html
[2] 
https://cgit.git.savannah.gnu.org/cgit/mailutils.git/commit/?id=be30644d0d1e1a2098413e6991e134447bab5703
[3] 
https://git.gnu.org.ua/mailutils.git/commit/?id=be30644d0d1e1a2098413e6991e134447bab5703