[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [bug-mailutils] Large file support in Mailutils
From: |
Kostas Zorbadelos |
Subject: |
Re: [bug-mailutils] Large file support in Mailutils |
Date: |
Fri, 23 Dec 2005 11:19:25 +0200 |
User-agent: |
Mutt/1.5.9i |
On Thu, Dec 22, 2005 at 02:46:51PM -0500, Alain Magloire wrote:
>
>
> > Hello,
> >
> > > First of all I'd like to announce that our custom mail.local based on
> > > GNU Mailutils is operational on our production mail servers. Our mail
> > > servers have about 200K users and we deliver ~1-1.5 million messages
> > > to mailboxes daily.
> >
> > Great.
> >
>
> Yes.
>
> > > I compiled both my mda and GNU Mailutils (snapshot
> > > 20051102) with -D_FILE_OFFSET_BITS=64. However this by itself was not
> > > enough. I investigated the issue and the problem was the use of fseek(3)
> > > instead of feeko(3), so I substituted all occurences of fseek() with
> > > feeko() in Mailutils sources (found in
> > > ./mailbox/file_stream.c,./lib/getpass.c,
> > > ./mimeview/mimetypes-gram.c,./mimeview/mimetypes.y). This was enough
> > > to fix the issues observed and have large file support.
> > [..]
> > > So I ask for these changes to be included upstream. Of course I am not
> > > sure if this is enough to cover all possible issues involved.
> >
> > Of course we are aiming to supporting large files in mailutils. However,
> > simply changing fseek to fseeko and ftell to ftello is not enough in
> > general. There are more problems involved. Most probably these do not
> > affect your installation, however they should be solved before releasing
> > the package:
> >
Hmm, now that I think of it, I have not changed occurences of ftell()
to ftello() as well. But in my setup, I see no errors.
>
> Glad to see, it worked for you, this give us a good hint that we did not
> rely too much on platform dependent behavior, thanks!
>
> > 1. Using fseeko creates portability problems. It can, however, be solved
> > using appropriate autoconf magic.
> >
>
> Agreed, we need to pass the right flags to the compiler, is the OS support
> this. Question is not fseeko() and friends part of Posix now? Or whatever
> XOpen standard?
>
To the UNIX systems I could check (GNU/Linux, [Free|Net|Open]BSD,
Solaris) fseeko() exists in the standard C library and works the same
way.
The NetBSD and OpenBSD man pages state that
"The fseeko() and ftello() functions conform to X/Open System
Interfaces and Headers Issue 5 (``XSH5'')."
while the FreeBSD page states
"The fseeko() and ftello() functions conform to IEEE Std 1003.1-2001
(``POSIX.1'')."
>
> > 2. The major problem is the use of off_t in the libmailutils. This data
> > type has different sizes depending on whether the library was
> > compiled with large file support or not. Thus, if one compiles
> > libmailutils with large file support and then links it to the program
> > that is compiled without that support, any mailutils functions taking
> > off_t as an argument will get incorrect data, which will lead, *in
> > the best case*, to a coredump. In the worst case it will trash whatever
> > user files involved, as well.
> >
I learned this the hard way, though I believe the problem is when an
off_t is returned from a mailutils function to the program (64-bit
value to fit in a 32-bit one). Anyway, both the program and the library
should be compiled with the same D_FILE_OFFSET_BITS settings.
> > Compiling two variants of libmailutils (with and without large file
> > support) is an awkward solution. Therefore I have introduced
> > (2005-11-15) a new data type `mu_off_t' which is equivalent to 64-bit
> > off_t and have changed all related functions and datatypes
> > accordingly. This makes libmailutils safe to use no matter in what
> > combination it is linked, although this creates a small overhead in
> > the usual 32-bit mode.
> >
>
> It sounds good to me. So basically if I read you right, you proposing to
> always use 64 bits i.e. mu_off_t ==> 64 bits.
>
That means you promote a 32 bit off_t to a 64 bit mu_off_t each time
and you use everywhere in mailutils code an mu_off_t, right?
This is an elegant cross-platform solution.
>
> > 3. Most of the libmailutils code formats off_t for printing
> > using "%lu" format directive, which does not work for 64-bit data
> > types. Using "%llu" is impossible due to portability problems. The
> > recent snapshot (mailutils-20051216.tar.gz) solves this problem as
> > well. However, these changes are not committed yet, because I do not
> > quite like the way I implemented them. I am searching for a more
> > elegant solution.
> >
>
> Then the question is: where do we actually need to print/format mu_off_t?
> On the top of my head, I do not remember any occurrences, so it may be a
> non-issue. And in the event that we need to print/format is it safe to
> downcast it to an off_t? The other possibility is simply to always use our
> auxiliary functions like asnprintf() that supports the format and to pass
> the buffer down to printf() and all for printing.
>
>
I am glad you already address the issues for LFS support. I would
really like to have a buildable snapshot when the changes are commited
to the repository.
Thanks again for everything.
Kostas
--
Kostas Zorbadelos
address@hidden contact: kzorba (at) otenet.gr
Out there in the darkness, out there in the night
out there in the starlight, one soul burns brighter
than a thousand suns.