Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 03 Jan 2008 00:50:56 +0100 Michael Albinus <address@hidden> wrote: 

MA> Ted Zlatanov <address@hidden> writes:
MA> I'm a little bit lost. Does it mean you don't want to offer *all*
MA> messages in a mailbox?
>> 
>> Correct, otherwise it's hard to handle invalid messages: are they
>> invalid files or something else?  I wanted also to treat duplicates as
>> automatic backups but if you don't like that idea then I'll drop it.

MA> Maybe we should see real life examples. I don't know whether it is
MA> always good to present selected contents only. If there are technical
MA> restrictions - that's another game.

(I'm leaving Ding on the CC in case anyone has comments)

Say you have these article UIDs and subjects, and UIDs are ordered by
date.  Note the ENCODED_FILENAME will be the filename encoded in some
way and /a/b/c is the filename without any risky characters; Gnus has
the subject encoding functionality already.  The subject format is
specifically to allow easy searching.

1 "tramp-imap-marker /a/b/c ENCODED_ABC"
2 "tramp-imap-marker /a/b/c ENCODED_ABC"
3 "tramp-imap-marker /a/b/c ENCODED_ABC"
4 "hello world"

Do we 

A) allow for three identically named files, dynamically renaming 2 and 3
with the Emacs <1> and <2> convention?

B) ignore 1 and 2, never create them intentionally, and (optionally)
erase duplicates when a file is saved?  This is probably easiest to
implement in code.

C) treat 1 and 2 as backups (there will be a tramp-imap-backups
parameter to control how many are allowed per filename, and older ones
over that limit are erased when a file is saved)?  We'll need a
mechanism to revert to a backup, which I don't think Tramp has built-in
at the moment.

Second question: is message 4 ignored?  I would prefer to do so, to
allow coexistence of tramp-imap.el with other messages (or even later
versions of tramp-imap.el).

I think it's fine to use message UIDs as the true file name, but we need
to decide what to do in the cases above.

Third question: namespaces.  I feel that it's much better
for the user to store all the files in a single mailbox:

INBOX.storage holds /a/b/c, /d/e/f, and /g/h/i

I believe you proposed that instead we should auto-create:

INBOX.storage.a.b to hold /a/b/c
INBOX.storage.d.e to hold /d/e/f
INBOX.storage.g.h to hold /g/h/i

Did I understand you correctly?  Maybe we could do both, allowing for a
"root mailbox" and a "root prefix".  In the first case, those would be

INBOX.storage /

and in the second case, they would be

INBOX.storage.a.b /a/b
INBOX.storage.d.e /d/e
INBOX.storage.g.h /g/h

And maybe the user can configure those mappings exactly.  I still think
it's too complicated for 99% of the users and they'll never need more
than one mailbox per virtual filesystem, but if you disagree I can do
the extra work.

I'll assume we've picked the single mailbox approach for the rest of the
message.  The implementations will change quite a bit as far as
directories are concerned if we use multiple mailboxes.

MA> As starting point you might look at tramp-smb.el or tramp-gw.el. Both
MA> are addons, like tramp-imap.el could be. tramp-fish.el might be examined
MA> as well, but this method isn't used anywhere I believe - it was merely a
MA> proof of concept I didn't want to throw away. And I never ever got a bug
MA> report about ...

OK, I'll look.  tramp-gw.el doesn't have any Emacs primitives
implemented at first glance, so I'll look at tramp-smb.el which defines
all those mappings nicely in tramp-smb-file-name-handler-alist.  That
list has quite a few methods; are they all necessary or do the default
handlers for some of them suffice?

Here's the list (omitting the commented-out items from tramp-smb.el)
with some questions and comments.  Sorry if most of this is obvious,
this is my first time diving into the Tramp code.  What function would
handle the truename-to-visualname translation?

add-name-to-file: could be a special "link message" or just a copy, like in 
Windows
copy-file: implemented as an APPEND
delete-directory: implemented with a search+delete for all matching messages
delete-file: search+delete of all matching messages
directory-file-name: tramp-handle-directory-file-name?
directory-files: search for matches
directory-files-and-attributes: search for matches, attributes always 777
dired-call-process: ignore
dired-compress-file: ignore
file-accessible-directory-p: always t
file-attributes: always 777
file-directory-p: needs a search, but we could have a file name that conflicts 
with a directory name
file-executable-p: always nil
file-exists-p: needs a search
file-local-copy: ?
file-remote-p: tramp-handle-file-remote-p
file-modes: tramp-handle-file-modes or hard-code
file-name-all-completions: needs a search
file-name-completion: tramp-handle-file-name-completion
file-name-directory: tramp-handle-file-name-directory
file-name-nondirectory: tramp-handle-file-name-nondirectory
file-newer-than-file-p: needs a search, plus backups may matter
file-ownership-preserved-p: ignore
file-readable-p: always t
file-regular-p: always t
file-symlink-p: always nil
file-truename: returns UID
file-writable-p: always t
find-backup-file-name: we need to decide
insert-directory: ?
insert-file-contents: search+retrieve
load: tramp-handle-load
make-directory, make-directory-internal: we need to decide what this should do
make-symbolic-link: ignore or implement as a special message
rename-file: append with new subject, delete original and its backups if
we decide to do backups
set-file-modes: ignore
set-visited-file-modtime: we could use a header for this
shell-command: ignore?
substitute-in-file-name: ?
unhandled-file-name-directory: tramp-handle-unhandled-file-name-directory?
vc-registered: always nil
verify-visited-file-modtime: ?
write-region: needs to do an append+delete of original+backups as needed; IMAP 
can't rewrite a message

Thanks
Ted