Re: lynx-dev on caching (long)

[Leonid Pauzner]

> There are two places where we may look for guidance:
>  * The HTTP protocol itself.
>  * Newer versions of libwww.

> Since HTTP is engineered to support cacheing, and is quite detailed about it,
> it may tell us a lot about what kinds of requests a level of a cache
> hierarchy can make - what kinds of parameters are needed, what modes of
> operation one could think of etc.  With necessary modifications of course,
> for the program-internal rather than over-the-network case.  Note that
> HTTP caching is completely based on the style B communication (above).
> (Squid caches talking ICP with each other are something else.)

> In libwww 5.2, I find in <http://www.w3.org/Library/src/HTReq.html>:

Well, I am highly impressed with libwww 5.2,
it was completely rewritten and now the code clean and well documented
(I especially like User/Architecture/*.html),
it also include DNS cache and a persistent document's cache.
I think both caches should be a system-wide
on multitask/multiuser system (it is not an user-agent job,
otherwise we got obvious wasting or resources: different user-agents,
different versions, etc.), all we need is a "history" calls
without any revalidation, think there is no such standard
calls for arbitrary system-wide cache.
So we need a user-agent's feature:
we already have rendered document's cache (in-memory)
and also need in-memory cache for sources for "mode 2" calls only.

And yes, it may be useful to have a persistent cache for "single user"
machines as optional (mainly for non-UNIX ports).

I am not optimistic in moving Lynx to libwww 5.2 -
although they are mutually similar there are lots of differences
(chartrans implementation, for example), this is a long-standing work
for guru. Also new libwww have a different documentations/comments style:
lynx's code looks hairy but self-documented (some how).

Is it possible to implement a _part_ of new libwww,
its HTTP/1.1 core including cache ?

> Currently the caching-related request vocabulary already consists of a bunch
> of flags: `reloading', `LYforce_no_cache', `LYoverride_no_cache',
> `LYinternal_flag', ..., text->no_cache, anchor->no_cache, in conjunction with
> various other bits and pieces.  This is very confusing.  These are all needed
> so HTLoadDocument() (& friends) can make a decision what to do.  Now imagine
> what happens if we add another cache level.  Or just modify or enhance the
> current cache's behavior.

> Again, putting more of the decision making in mainloop() may seem an easy way
> out; I think instead there is an urgent need to rationalise the language,
> before much gets added cachewise.

>   ------------ snip ------------
> HTTP Cache Validation and Cache Control

>    The Library has two concepts of caching: in memory and on file. When 
> loading a document,
>    this flag can be set in order to define who can give a response to the 
> request. The
>    mempory buffer is considered to be equivalent to a history buffer. That 
> is, it doesn't not
>    follow the same expiration mechanism that is characteristic for a 
> persistent file cache.

>    You can also set the cache to run in disconnected mode - see the [23]Cache 
> manager for
>    more details on how to do this.
> typedef enum _HTReload {
>     HT_CACHE_OK             = 0x0,              /* Use any version available 
> */
>     HT_CACHE_FLUSH_MEM      = 0x1,      /* Reload from file cache or network 
> */
>     HT_CACHE_VALIDATE       = 0x2,                   /* Validate cache entry 
> */
>     HT_CACHE_END_VALIDATE   = 0x4,                  /* End to end validation 
> */
>     HT_CACHE_RANGE_VALIDATE = 0x8,
>     HT_CACHE_FLUSH          = 0x10,                     /* Force full reload 
> */
>     HT_CACHE_ERROR          = 0x20         /* An error occurred in the cache 
> */
> } HTReload;

> extern void HTRequest_setReloadMode (HTRequest *request, HTReload mode);
> extern HTReload HTRequest_reloadMode (HTRequest *request);

> ------------ snip ------------

> He, that looks just like what we need!  Instead of setting many variables
> mainloop() and other higher-level functions could just set one request
> variable to the right value, using max() if appropriate, to bump up the,
> hmm, nocache-nature of a request.

> Rough correspondence to current flags (omitting the two that may not really
> belong here):
>     HT_CACHE_OK             default or LYoverride_no_cache
>     HT_CACHE_FLUSH_MEM      LYforce_no_cache, text->nocache
>     HT_CACHE_VALIDATE
>     HT_CACHE_END_VALIDATE
>     HT_CACHE_FLUSH          `reloading'

> Well, the correspondence isn't a direct one. But I think something like
> this should replace the current flags as far as possible.  With added
> values if we need more.