bug-mailutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [bug-mailutils] Large file support in Mailutils


From: Kostas Zorbadelos
Subject: Re: [bug-mailutils] Large file support in Mailutils
Date: Fri, 23 Dec 2005 11:19:25 +0200
User-agent: Mutt/1.5.9i

On Thu, Dec 22, 2005 at 02:46:51PM -0500, Alain Magloire wrote:
> 
> 
> > Hello,
> > 
> > > First of all I'd like to announce that our custom mail.local based on
> > > GNU Mailutils is operational on our production mail servers. Our mail
> > > servers have about 200K users and we deliver ~1-1.5 million messages
> > > to mailboxes daily.
> > 
> > Great.
> > 
> 
> Yes.
> 
> > > I compiled both my mda and GNU Mailutils (snapshot
> > > 20051102) with -D_FILE_OFFSET_BITS=64. However this by itself was not
> > > enough. I investigated the issue and the problem was the use of fseek(3)
> > > instead of feeko(3), so I substituted all occurences of fseek() with
> > > feeko() in Mailutils sources (found in
> > > ./mailbox/file_stream.c,./lib/getpass.c,
> > > ./mimeview/mimetypes-gram.c,./mimeview/mimetypes.y). This was enough
> > > to fix the issues observed and have large file support.
> > [..]
> > > So I ask for these changes to be included upstream. Of course I am not
> > > sure if this is enough to cover all possible issues involved.
> > 
> > Of course we are aiming to supporting large files in mailutils. However,
> > simply changing fseek to fseeko and ftell to ftello is not enough in
> > general. There are more problems involved. Most probably these do not
> > affect your installation, however they should be solved before releasing
> > the package:
> >

Hmm, now that I think of it, I have not changed occurences of ftell()
to ftello() as well. But in my setup, I see no errors.

> 
> Glad to see, it worked for you,  this give us a good hint that we did not
> rely too much on platform dependent behavior, thanks!
>  
> > 1. Using fseeko creates portability problems. It can, however, be solved
> >    using appropriate autoconf magic.
> >
> 
> Agreed, we need to pass the right flags to the compiler, is the OS support
> this.  Question is not fseeko() and friends part of Posix now? Or whatever
> XOpen standard?
> 

To the UNIX systems I could check (GNU/Linux, [Free|Net|Open]BSD,
Solaris) fseeko() exists in the standard C library and works the same
way. 

The NetBSD and OpenBSD man pages state that
"The fseeko() and ftello() functions conform to X/Open System
Interfaces and Headers Issue 5 (``XSH5'')." 

while the FreeBSD page states
"The fseeko() and ftello() functions conform to IEEE Std 1003.1-2001
(``POSIX.1'')."

>  
> > 2. The major problem is the use of off_t in the libmailutils. This data
> >    type has different sizes depending on whether the library was
> >    compiled with large file support or not. Thus, if one compiles
> >    libmailutils with large file support and then links it to the program
> >    that is compiled without that support, any mailutils functions taking
> >    off_t as an argument will get incorrect data, which will lead, *in
> >    the best case*, to a coredump. In the worst case it will trash whatever
> >    user files involved, as well.
> > 

I learned this the hard way, though I believe the problem is when an
off_t is returned from a mailutils function to the program (64-bit
value to fit in a 32-bit one). Anyway, both the program and the library
should be compiled with the same D_FILE_OFFSET_BITS settings.

> >    Compiling two variants of libmailutils (with and without large file
> >    support) is an awkward solution. Therefore I have introduced
> >    (2005-11-15) a new data type `mu_off_t' which is equivalent to 64-bit
> >    off_t and have changed all related functions and datatypes
> >    accordingly. This makes libmailutils safe to use no matter in what
> >    combination it is linked, although this creates a small overhead in
> >    the usual 32-bit mode.
> >
> 
> It sounds good to me.  So basically if I read you right, you proposing to
> always use 64 bits i.e. mu_off_t ==> 64 bits.
>

That means you promote a 32 bit off_t to a 64 bit mu_off_t each time
and you use everywhere in mailutils code an mu_off_t, right?
This is an elegant cross-platform solution.
 
>  
> > 3. Most of the libmailutils code formats off_t for printing
> >    using "%lu" format directive, which does not work for 64-bit data
> >    types. Using "%llu" is impossible due to portability problems. The
> >    recent snapshot (mailutils-20051216.tar.gz) solves this problem as
> >    well. However, these changes are not committed yet, because I do not
> >    quite like the way I implemented them. I am searching for a more
> >    elegant solution.
> > 
> 
> Then the question is: where do we actually need to print/format mu_off_t?
> On the top of my head, I do not remember any occurrences, so it may be a
> non-issue.  And in the event that we need to print/format is it safe to
> downcast it to an off_t? The other possibility is simply to always use our
> auxiliary functions like asnprintf() that supports the format and to pass
> the buffer down to printf() and all for printing.
> 
> 

I am glad you already address the issues for LFS support. I would
really like to have a buildable snapshot when the changes are commited
to the repository.

Thanks again for everything.
Kostas 

-- 
  Kostas Zorbadelos
  address@hidden contact: kzorba (at) otenet.gr
  
  Out there in the darkness, out there in the night
  out there in the starlight, one soul burns brighter
  than a thousand suns.





reply via email to

[Prev in Thread] Current Thread [Next in Thread]