Hideously slow VC status queries fixed

[Eric S. Raymond]

It gives me great pleasure to be able to announce that I have found,
and fixed, the bug that made C-x v d so godawful much slower than the
underlying commands.

This one was a truly classic collision of blunders.  It should be a
warning to us all about the perils of the expedient local hack.

When I originally wrote VC mode back in the dark and backward abysm of
time, it was for RCS and SCCS.  Back then there was no dir-state hook;
vc-dired-hook queried the version-control status of each file named in
the VC-dired buffer individually.  This performed acceptably because
code directories averaged far smaller then.

Whoever wrote the first dir-state hook (probably for CVS) blundered
badly by making the most conservative possible modification to
vc-dired.  That unknown hacker, may his name be accursed, used
dir-state *only for subdirectories*; the status of all files in
default-directory was still queried one at a time.  He carefully wrote
the dir-state hook to use "status -l", not recursing down
subdirectories, because he gathered all the subdirectories by walking
through the dired listing and applied the dir-state method to each
one at a time.

People who wrote backends for later VCSes followed suit, so focused
on getting their backends minimally working that they failed to notice
that the upper-level logic was perversely misdesigned. 

And this threw away a lot of data; most of the later back ends are
like SVN in there's no way to tell the status command not to recurse
down directories.  So the status information for the *entire file
tree* would get queried and parsed as many times as there were
non-excluded directory nodes in the tree.

Gaaaahhhhh...no *wonder* it was slower than a snail on Quaaludes!

All I did to fix this was notice that you *want* dir-state hook to
recurse down directory tries -- and then call it exactly once on
default-directory.  Most of the fix consisted of removing options
to suppress the recursion and documenting the new expected behavior
of dir-state.

Someone should have noticed this a lot sooner.
-- 
                <a href="http://www.catb.org/~esr/";>Eric S. Raymond</a>

What, then is law [government]? It is the collective organization of
the individual right to lawful defense."
        -- Frederic Bastiat, "The Law"