Re: [bug-mailutils] Large file support in Mailutils

[Sergey Poznyakoff]

Hello,

> First of all I'd like to announce that our custom mail.local based on
> GNU Mailutils is operational on our production mail servers. Our mail
> servers have about 200K users and we deliver ~1-1.5 million messages
> to mailboxes daily.

Great.

> I compiled both my mda and GNU Mailutils (snapshot
> 20051102) with -D_FILE_OFFSET_BITS=64. However this by itself was not
> enough. I investigated the issue and the problem was the use of fseek(3)
> instead of feeko(3), so I substituted all occurences of fseek() with
> feeko() in Mailutils sources (found in
> ./mailbox/file_stream.c,./lib/getpass.c,
> ./mimeview/mimetypes-gram.c,./mimeview/mimetypes.y). This was enough
> to fix the issues observed and have large file support.
[..]
> So I ask for these changes to be included upstream. Of course I am not
> sure if this is enough to cover all possible issues involved.

Of course we are aiming to supporting large files in mailutils. However,
simply changing fseek to fseeko and ftell to ftello is not enough in
general. There are more problems involved. Most probably these do not
affect your installation, however they should be solved before releasing
the package:

1. Using fseeko creates portability problems. It can, however, be solved
   using appropriate autoconf magic.
   
2. The major problem is the use of off_t in the libmailutils. This data
   type has different sizes depending on whether the library was
   compiled with large file support or not. Thus, if one compiles
   libmailutils with large file support and then links it to the program
   that is compiled without that support, any mailutils functions taking
   off_t as an argument will get incorrect data, which will lead, *in
   the best case*, to a coredump. In the worst case it will trash whatever
   user files involved, as well.

   Compiling two variants of libmailutils (with and without large file
   support) is an awkward solution. Therefore I have introduced
   (2005-11-15) a new data type `mu_off_t' which is equivalent to 64-bit
   off_t and have changed all related functions and datatypes
   accordingly. This makes libmailutils safe to use no matter in what
   combination it is linked, although this creates a small overhead in
   the usual 32-bit mode.

3. Most of the libmailutils code formats off_t for printing
   using "%lu" format directive, which does not work for 64-bit data
   types. Using "%llu" is impossible due to portability problems. The
   recent snapshot (mailutils-20051216.tar.gz) solves this problem as
   well. However, these changes are not committed yet, because I do not
   quite like the way I implemented them. I am searching for a more
   elegant solution.

Regards,
Sergey