Re: [Monotone-devel] slow update/status on NFS

[Nathaniel Smith]

On Wed, May 23, 2007 at 02:13:56PM +0200, Markus Schiltknecht wrote:
> Humpf... I've just added a 'path::status  cached_status;' to any_path, 
> with an additional which returns that or gets the status.
> 
> The problem now is, that this routine needs to *write* the status into 
> any_path.cached_status, requiring that the any_path in question is not 
> const, but writable. That breaks a whole lot of any_path & const xy 
> routines. And I'm unsure if its good to remove all those consts.

Probably not.

> Another way to do it would be, implementing all that caching inside the 
> platform dependent 'get_path_status'. But on second thought, this isn't 
> that simple as well, because we don't know about the lifetime of the 
> underlying any_path class there.

Caching is _never_ simple.  Aside from such implementational issues,
the idea of a cache is that you take a system that has complex dynamic
behavior you can't analyze very well (or else you'd just optimize it),
bolt another system with its own complex dynamic behavior (expiration
policies, blah blah blah) onto the side, and hope that the two of them
interact in a nice way so that efficiency gains emerge.  Which isn't
to say that caching is bad, it's often very good, just it's uses
should *mostly* be confined to places where you do not know or control
the layer above you whose accesses you are trying to optimize.

The exception is when you know the layer above you perfectly well, but
it's complicated enough that you either can't figure out how to do
something smarter than straight caching, or are too lazy to do
something smarter right now.  (We have rather a few of these sorts,
actually, e.g. all the db caching stuff...)

In the directory scanning case, though, we've basically got a single
straight-line hunk of code that is run exactly the same way over and
over in an inner loop.  Maybe we should think about just slimming it
down instead of adding new mechanisms?

To start with, there's absolutely no need to check that a path is a
file before opening it for reading -- if it's not a file, then the
read will fail just fine on its own.  I threw that assertion in
because at the time I figured if we were already reading the whole
file than a stat was too cheap to worry about, but if this is wrong
then let's take that out and handle errors from read.

-- Nathaniel

-- 
Details are all that matters; God dwells there, and you never get to
see Him if you don't struggle to get them right. -- Stephen Jay Gould