Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Sat, Jun 23, 2018 at 2:24 AM, Eli Zaretskii <address@hidden> wrote:
>> From: Alex Harsanyi <address@hidden>
>> Date: Fri, 22 Jun 2018 20:46:59 +0800
>> Cc: address@hidden
>>
>> > Does it help to set w32-pipe-buffer-size to some large value, like
>> > 16384, and/or decrease w32-pipe-read-delay?
>>
>> If I set w32-pipe-read-delay to 0 the speed improvement is
>> significant.  On top of that, there is an additional speed
>> improvement if I set w32-pipe-buffer-size to 16384
>
> Can you show numbers for these speedups, and compare the times with
> Emacs 25.  It would also be useful if you show times with different
> values of these, so that the optimal values could be chosen.
>
> I'd be also interested in the timings with the larger repository.

I tested various values for `w32-pipe-read-delay` on my work git repository.
The original timings that I made were against a work tree that had several
modified files and I used a stop watch to time the update.  These tests were
run on a large (51678 tracked files) but clean working directory, and I
modified the source files to time the commands using `current-time`, so I got
more precise timings.

    $ git ls-files . | wc -l
    51678

    $ git status
    On branch ah_tmp
    nothing to commit, working directory clean

Emacs 26.1 runs the following commands as part of a `vc-dir` command, or an
update of the `*vc-dir*` buffer (Emacs 25.2 runs the same commands, except it
does not run the third one)

    * git --no-pager update-index --refresh .
    * git --no-pager diff-index --relative -z -M HEAD -- .
    * git --no-pager ls-files -z -c -s -- .
    * git --no-pager ls-files -z -o --directory --no-empty-directory
--exclude-standard -- .

On a clean working directory, only "git --no-pager ls-files -z -c -s -- ."
produces any output, but produces 6.3 Mb of it.  The command itself runs fast:

    $ git --no-pager ls-files -z -c -s -- . | wc
          0  184463 6689336

    $ time git --no-pager ls-files -z -c -s -- . > /dev/null

    real    0m0.619s
    user    0m0.000s
    sys     0m0.015s

I tried several values for `w32-pipe-read-delay` to see how long it takes to
run these commands in Emacs 26.1:

`w32-pipe-read-delay` is set to 50 milliseconds (default for Emacs 26.1)

    Running git --no-pager update-index --refresh . in background...
done 0.26 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.34 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 83.70 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.64 seconds

`w32-pipe-read-delay` set to 25 milliseconds

    Running git --no-pager update-index --refresh . in background...
done 0.26 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.34 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 42.77 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.67 seconds

`w32-pipe-read-delay` set to 12 milliseconds

    Running git --no-pager update-index --refresh . in background...
done 0.27 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.35 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 21.57 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.65 seconds

`w32-pipe-read-delay` set to 0 milliseconds (no delay)

    Running git --no-pager update-index --refresh . in background...
done 0.31 seconds
    Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.39 seconds
    Running git --no-pager ls-files -z -c -s -- . in background...
done 1.07 seconds
    Running git --no-pager ls-files -z -o --directory
--no-empty-directory --exclude-standard -- . in background... done
0.69 seconds

So, with a 50 millisecond delay, it takes 83.7 seconds to read the command
output and this is reduced to 1.07 seconds when the delay is set to 0.

Alex.