gnats/665: Re: gnats/665: queue-pr -r doesn't process the incoming messages in the right order

[bug-gnats]

The following reply was made to PR gnats/665; it has been noted by GNATS.

From: Stephane Chazelas <address@hidden>
To: address@hidden
Cc: address@hidden, address@hidden
Subject: Re: gnats/665: queue-pr -r doesn't process the incoming messages in 
the right order
Date: Wed, 9 Aug 2006 18:30:48 +0100

 On Wed, Aug 09, 2006 at 12:15:01PM -0500, address@hidden wrote:
 [...]
 >  address@hidden  wrote:
 >  >  I guess you refer to the arbitrary "2000" limit on the size of the
 >  >  incoming queue. Is there any other "magic number" you're thinking
 >  >  of?
 >  
 >  That's one, and the other is the character array you allocated for the
 >  file name.  32 characters doesn't necessarily translate to a 32 bit
 >  number.  Potentially, that would be your overflow point with the
 >  number of files allowed in the queue if file names are going to be
 >  sequential.
 
 32 only meant to be something greater than the length of an
 integer number. a 31 character large number could only be issued
 by a sprintf on a 128 bit system, and such a number would be
 that big that no file system on earch in the coming millenium
 could hold that many files. Feel free to increment it or use
 asprintf instead, or something like
 char filename[] = "X" #INT_MAX;
 
 >  A more scalable solution would be to use a Maildir style
 >  naming convention, where date-time when the file was received in
 >  combination with the host name and and a delivery identifier is used.
 >  
 >  See: http://cr.yp.to/proto/maildir.html
 >  
 >  This is actually a very good method for queues received asynchronously
 >  on different machines.  For example, if someone is using gnats on an
 >  NFS share, this naming style would allow it to happen without locking
 >  and without danger of name collision.
 >  
 >  Incorporating this into a queue processing mechanism would allow you
 >  to use the "unique:info" pair to rename the incoming queue files to an
 >  info flag such as "P" for processing, "E" for error.  queue-pr could
 >  be told to simply ignore both and only process those files w/o the
 >  info tag appended to it.  That way,  you wouldn't have to worry about
 >  two queue-pr's running at the same time...  Unless there is
 >  database-locking semantics to look out for.  I'll dig into that more.
 
 It's true, though it may be a bit overkill. As far as I can tell
 the solution I came up with is also scalable and allows several
 writers and knows how to deal with several readers.
 
 But if there's already existing code you can borrow to implement
 the maildir algorithm, that may be a better approach.
 
 [...]
 >  >  The idea was to make sure those kind of problems were not left
 >  >  unnoticed, but you're right that they will get noticed when the
 >  >  file system or the directory is full or when it takes so much time
 >  >  to add an entry to a directory that several instances of queue-pr
 >  >  -q run at the same time.
 >  
 >  Right.  I'm going to try to dig into it a bit more today if I can.
 [...]
 
 Just remove the if (i == 2000) exit and that should be fine.
 
 Cheers,
 Stephane