Re: A question

[Alain Magloire]

Bonjour

CC'ed: to mailutils.

> 
> >I would recommand to provide a threshold, for example 
> >caching upto XXX Kbytes meaning if the threshold is reach 
> >part of the buffer should be flush away and reset etc .. 
> >this will make the code a bit harder but save in term
> >of memory when it is desired.
> 
> Yes... I was thinking about that when I recieved this. I have a lower
> limit on fetch sizes because there's obviously going to be a lower
> limit to the economy of doing this (I'll probably resolve commands
> that cause partial fetches of the whole message to be turned into
> normal fetches).
> 
> 
> >in the case of
> >  FETCH 1 (body[]) <0.5000>
> >The buffer will be reset 10 times for limit of 500.  For a limit
> >of 10000 it will still have room to grow.
> 
> I was thinking about a buffer of 5Kb but I want the buffer to be
> configurable so I'm coding for that.
> 
> I could add code to very intelligently handle the buffering, but I
> don't think I will. The buffer will fill to the maximum size and then
> no further underlying-stream read will be performed. The buffer will
> always empty before more underlying-stream reads are performed.

Yes, buffering can be tricky, especially if read/write are allowed, it
can also be a bit tricky if offsets are involve.

Another point may or may not be important to you, is the offset is
given in term of RFC822, for IMAP4 and the propose draft RETR-offset
for POP3.

In our case(GNU mailutils), when coding part of the imap4d(still work
in progress) the translation is/was really annoying.  Since, in GNU
mailutils, the stream_t return by the message is given in term of the
native Unix format i.e. "\n".  IMAP servers want the offset in term of
RFC822 "\r\n" terminated line.
So for example, when a client ask to read form offset 10, 500 bytes:
stream_read (stream, buffer, 500, 10, NULL);

This will map to IMAP like

FETCH 1 BODY[]<15.500>

10 --> 15 is the mapping.

> Since the stream is designed to be used in the webmail environment by
> an app that has set the fetch size this should be efficient. Here's an
> elaboration of the java code I sent in the last message, this is real
> code that is used in one of the webmail tools I work on:
> 
>   Message m=folder.getMessage(1);
>   InputStream in=message.getRawInputStream();
>   byte[] buf=new byte[5000];
>   int red=in.read(buf,0,5000);
>   while(read>-1)
>    {
>       out.write(buf,0,red);
>       red=in.read(buf,0,5000);
>    }
> 
> "out" is the HTTP response output stream where the data is being
> written. This is a very usefull code snippet when the message is an
> image or an HTML file or something that can be returned direct to the
> browser.
> 
> The addition of this "on demand" system should make things even more
> efficient.
> 

Ok, I see where you are going.

> >Caching makes random access of the message 
> >with different offset easier.
> 
> I'm actually trying to do this to eliminate caching but I do see what
> you mean. Javamail provides that (usually) through caching of the
> entire message content. I will offer that as a runtime config option.
> 
> 
> >The way GNU mailutils will tackle this, is different, the 
> >two main ideas we are considering are:
> >- A caching mailbox, a mailbox that sits on top of another 
> >  one so all the requests are cache. For example:
> 
> I've been wanting to explore this for Javamail. One of the people who
> encouraged me to write this new provider in the first place was
> someone who wanted a "disconnected mode" of operation for his java
> based mail client. I thought at the time that a provider could do all
> the cahing he needed on some config switch. All the network
> connectivity headaches (eg: constantly trying connection) could be
> hidden. 
> 
> I still think this is true but I'm having trouble imagining how I'd
> swap in the caching layer into my current provider architecture. I
> think it means a lot of config detection in my base javamail objects.
> I've been thinking that perhaps an additional layer on top of my
> current stuff might do the job.

8-)
Sometimes, a piece of the puzle, will just not find a place 8-)
meaning it will not fit easily with what you have so far, grrrr!!

> >I think it should be possible 
> >  mailbox_t mbox;
> >  mailbox_create (&mbox, "cache:pop://localhost");
> >  so the requests are pass to the pop://localhost mailbox 
> >  but the results are cache by the caching"cache:" mailbox.
> >  The cache mailbox can also give persistency by saving the 
> >  results to files. Since the caching is abstract :
> >    mailbox_create (&mbox, "cache:imap://localhost/INBOX");
> >  etc ..
> 
> Agreed. This *should* be very easy. 
> 
> In javamail caching like this must be done by the provider since the
> application cannot know what is being cached and cannot properly
> construct a mailbox from externally cached data. In other
> architectures though I'm not sure this is worthwhile. Is it usefull to
> you just as an abstraction of IMAP?

In our framework, it is usefull for POP.  POP3 does not provide random
access to a message etc .. by letting the caching mailbox do the dirty
work, the client can treat the mailbox as a normal local mailbox.

Our stream_t object does not provide much caching,(well it does but
it is very basic), by doing two reads:
offset = 0;
stream_read (stream, buffer, sizeof (buffer), offset, ..);
offset = 300;
stream_read (stream, buffer, sizeof (buffer), offset, ..);

This will map to two network access(FETCH commands).  Moving the caching
complexity outside to a special mailbox help to ease the code maintainance.

The problems will arise when we will want persistency of the "disconnected
mode".  For example doing a partial fetch, close the mail browser, come back
later restart the application redo the same partial fetch but since the
cache was save to disk no networking access is done.

Our caching will provide synchronisation and will have to come upe with some
sort of format to save the information to file(s).


> 
> >- Doing the caching of the stream:
> >  mailbox_t mbox;
> >  message_t msg;
> >  stream_t stream;
> >  mailbox_create (&mbox, "pop://localhost");
> >  mailbox_get_message (mbox, 2, &msg);
> >  message_get_stream (msg, &stream);
> >  stream_read (stream, buffer, sizeof (buffer), NULL);
> > ...
> >  By doing the stream_read (), everything will be cache.
> >- Having a special stream caching object, this follow what you
> >  were proposing above.
> 
> Yeah... I feel more comfortable with this approach outside of the
> "disconnected mode" idea. But I'm not really doing it so that a cache
> can be obtained, I'm doing it so that a cache can be avoided.

Agreed.

> What my stream allows me to do is pass through socket reads (of the
> IMAP connection) direct to the application without using a cache
> inside the Message representation. This is perfect for webmail since
> you don't want your webmail HTTP thread request invocations to have to
> read in an entire attachment before you can write it out as an HTTP
> response. HTTP works best with largish chunks of data being dropped
> onto the stream, that's why I'm using about 5Kb as a buffer minimum.
> 
> 
> 
> Nic
> 
> >Aussi haut que l'on soit assis, on est toujours 
> >assis que sur son cul !!!
> 
> I'm affraid as an english person I don't know much French. I'm going
> to try to learn soon.

Excellent!!


-- 
au revoir, alain
----
Aussi haut que l'on soit assis, on est toujours assis que sur son cul !!!