Re: lynx-dev dev.16 patch 3 - SOURCE_CACHE etc.

[Klaus Weide]

On Fri, 10 Dec 1999, Leonid Pauzner wrote:
> 10-Dec-99 07:41 Klaus Weide wrote:
> > On Fri, 10 Dec 1999, Leonid Pauzner wrote:
> 
> >> There is one fundamental difference between cache for "reparsing" and
> >> cache in HTTP sence we discussed before. Reparsing -just another
> >> presentation of html document- should be done on the same source, no
> >> cache-control or expire logic here at all. 

(But I see "no cache-control or expire logic" as just one (degenerate) case
of having "cache-control or expire logic".  I'ts just the most permissive
cache-control/expire policy.)

> >> That is why it was
> >> implemented in mainloop and not in getfile [which will require something
> >> like "disable expiration logic" in many places].
> 
> > Not in many places.  Just one flag should be needed to say "take this
> > from the souce cache if it is there, overriding all other concerns",
> > and changes to avoid HTuncache_current_document calls from mainloop
> > (or call a modified version HTuncache_but_dont_completely_forget).
> 
> You forget lots of places in mainloop, such as check a replay from POST,
> which only required when network transmition will be the case.

Yes, true.  But those could be deferred until a point when it becomes
clear that a network transmission would be necessary.   There is already
such a deferred prompt in HTLoadDocument:

    /*
     *  If this is marked as an internal link but we don't have the
     *  document loaded any more, and we haven't explicitly flagged
     *  that we want to reload with LYforce_no_cache, then something
     *  has disappeared from the cache when we expected it to be still
     *  there.  The user probably doesn't expect a new network access.
     *  So if we have POST data and safe is not set in the anchor,
     *  ask for confirmation, and fail if not granted. [...]
     */

> >> Instead, in
> >> the first case more mainloop changes welcome to make things more
> >> obvious, like curdoc.line!=Newline before HText_pageDisplay(): I mean
> >> current newline logic vs proper refresh_screen check where complete
> >> refresh is really assumed.
> 
> > I don't understand.  Are you saying that the changes to mainloop() to
> > support source_cache have made anything clearer?  Definitely not for
> > me.
> I was saying... but never get implemented mostly. sory.
> At least your from_source_cache_p will not made things more clear,
> that should be LYMainLoop.c PRIVATE until it will be completely removed
> one day (it was a fix for curdoc.line and www_search_result glitches
> anyway).

It wasn't clear to me that it is scheduled for removal.

> > You give too little detail for letting me understand what you mean wrt.
> > "proper refresh_screen check".
> 
> I mean two HText_pageDisplay() calls in mainloop(). It looks very
> strange for me to use  if (curdoc.line != Newline)
> as alternative for  if (refresh_screen). 

Well, according to the comment (quoted below) this is just a
roundabout way for testing whether there has "been a change".  This
covers the case when a new document has been loaded, as well as when
the "window into the document" has changed.  I don't know what else
exactly it covers, and/or is supposed to cover.  But yes, maybe
(curdoc.line != Newline) should be replaced by some more obvious
condition.  I just don't know what that would be, and whether it
wouldn't break something.

> It is not clear,
> we set refresh_screen on a lots of places in LYMainLoop.c
> and does not use it explicitely.

What do you mean with "does not use it explicitely"?
After all it *is* used - in the code you quote.

> Also I think Newline should be set to www_search_result in
> handle_LYK_WHEREIS() and in that "force_load" part to pick
> HTFindPoundSelector() result. Perhaps I will submit a patch.

I declare you the expert on Newline.  It has never been entirely clear
to me, even before all the SOURCE_CACHE changes.

>         /*
>          *  If the curdoc.line is different than Newline then there must
>          *  have been a change since last update.  Run HText_pageDisplay()
>          *  create a fresh screen of text out.
>          *  All display_partial calls ends here for final redraw.
>          */
>         if (curdoc.line != Newline) {
> #ifndef NO_NONSTICKY_INPUTS
>             textinput_drawn = FALSE;
> #endif
> 
>             refresh_screen = FALSE;
> 
>             HText_pageDisplay(Newline);
> ---------------------
> 
> and next after above nearly the same chunk of code starting with:
> 
>         /*
>          *  Refresh the screen if necessary.
>          */
>         if (refresh_screen) {
> #if defined(FANCY_CURSES) || defined (USE_SLANG)
>             if (enable_scrollback) {
>                 clear();
>             } else {
>                 erase();
>             }
> #else
>             clear();
> #endif /* FANCY_CURSES || USE_SLANG */
>             HText_pageDisplay(Newline);

What do you find wrong with this?
Do you want to combine the two blocks into one?  Note that there *are*
some differences.

> > You should be able to tweak that external cache so that it ignores
> > pre-expiration and even no-cache and always caches anyway.  Although
> > I haven't tried it in a while, I believe there are configuration options
> > for this in squid.
> We do not need "always caches anyway", at least responce from
> POST request for sure. We want it in very special situations only.

I doubt that it is even possible to cache POST responses with squid, or
another caching proxy.  The cache would have to store the post data in
addition to the Request-URL.  I doubt that any of the usual candidates
do that.

> > Also, if the file is cached with SOURCE_CACHE:FILE - it's already in
> > a file on your disk, you can access that file directly and do with it
> > what you want...  (You first have to figure out the filename though).
> Well, this is extra.
> > (We could display the cache filename on the INFO screen for folks who
> > really want to do this.)
> No. They could study the trace log if they are so interested...

Why are you so hostile to that idea? :)
I think showing the cache filename on the INFO screen (only in Advanced
mode) is not a bad idea.

> > If you are suggesting that 'd' *in the History Page* should act differently
> > and take a source_cache copy if available - but not 'd' on another link -
> > yeah I see some logic in that.  But that would of course only help for
> > text/html.
> 
> Yes. We starting to understand each other at this moment:)

Somewhat. :)

> While 'd'ownloading from the History page is a special case,
> recommended in Users Guide, it would be nice to utilize source_cache
> copy when we already have it.
> 
> BTW, we usually have *only* text/html documents in the History page,
> exceptions are rare (local/ftp directories and text/plain documents).
> No binaries.

No binaries, sure (except for mislabelled binaries that end up being
treated as text).  But whether it's rare or not depends on how you are
using Lynx.  If you are using it for NetNews reading, or for local
directory browsing, or for gopher, or FTP, or other things I can't
think of right now, non-HTML documents may not be rare.

   Klaus