Re: A project-files implementation for Git projects

[Tassilo Horn]

Dmitry Gutov <address@hidden> writes:

Hi Dmitry,

>> Ah, "hg status --all" lists all files including their state
>> (untracked, ignored, you-name-it), so that's the one we should use.
>> Performance seems to be the same as for "hg files".
>
> In my testing the performance difference is about 2x:
>
> $ bash -c "time hg status -c >/dev/null"
>
> real  0m12,015s
> user  0m1,899s
> sys   0m10,113s
>
> $ bash -c "time hg files >/dev/null"
>
> real  0m5,970s
> user  0m1,004s
> sys   0m4,965s
>
> (project-files (project-current)) takes ~7 seconds here on the same repo
> (Mozilla Firefox checkout).
>
> But if it's faster than 'find' anyway on some platforms, why not? As
> long as there's a solution that will handle the adjusted ignore rules
> in a similarly performant fashion.

Right.

>> I think we can come up with a VC list-files operation which
>> optionally includes untracked and ignored files (where the latter
>> implies the former, doesn't it?)
>
> Whether it implies or not, depends on which set of ignores we're
> talking about (Git's own or the modified one).
>
>> but I'd leave the filtering according to project-vc-ignores to
>> project.el.
>
> Have you tries benchmarking this approach? E.g. calling 'git ls-files
> -c -o -z' and then doing all the filtering indicated by .gitignore
> rules?
>
> Try it on the current Emacs repo.
>
> IME it's the ignore rules that take up 99% of the CPU time when using
> 'find'. Without them, 'find .' is instant (though that depends on the
> disk access speed). If we're going to implement that in Elisp, I'd
> wager it's going to be even slower.

Well, ok.  I've now played with an interface

  (vc-call-backend (vc-responsible-backend dir)
                   'list-files
                   dir
                   include-unregistered
                   extra-includes)

where extra-includes works in addition to the standard VC ignore rules
(.gitignore, .hgignore).  Or do you want to override the VC-internal
rules?

At least for Git and Hg, I came up with reasonable implementations:

--8<---------------cut here---------------start------------->8---
(defun vc-git-list-files (&optional dir
                                    include-unregistered
                                    extra-ignores)
  (let ((default-directory (or dir default-directory))
        (args '("-z")))
    (when include-unregistered
      (setq args (nconc args '("-c" "-o" "--exclude-standard"))))
    (when extra-ignores
      (setq args (nconc args
                        (mapcan
                         (lambda (i)
                           (list "--exclude" i))
                         (copy-list extra-ignores)))))
    (mapcar
     #'expand-file-name
     (cl-remove-if
      #'string-empty-p
      (split-string
       (apply #'vc-git--run-command-string nil "ls-files" args)
       "\0")))))

(defun vc-hg-list-files (&optional dir
                                   include-unregistered
                                   extra-ignores)
  (let ((default-directory (or dir default-directory))
        args
        files)
    (when include-unregistered
      (setq args (nconc args '("--all"))))
    (when extra-ignores
      (setq args (nconc args
                        (mapcan
                         (lambda (i)
                           (list "--exclude" i))
                         (copy-list extra-ignores)))))
    (with-temp-buffer
      (apply #'vc-hg-command t 0 "."
             "status" args)
      (goto-char (point-min))
      (while (re-search-forward "^[?C]\s+\\(.*\\)$" nil t)
        (setq files (cons (expand-file-name (match-string 1))
                          files))))
    (nreverse files)))
--8<---------------cut here---------------end--------------->8---

There's a semantic difference between Git and Hg in the treatment of
extra-ignores.  With Git, the extra-ignores do not rule out committed
files (i.e., they are only effective for untracked files) while for Hg,
they also rule out committed files.  I think the Hg semantics are
probably better but I don't see how to change the Git version so that it
acts the same way (except by re-filtering in lisp, of course), do you?

I haven't looked at the other backends.  I guess bzr will probably be
doable, too.  However, for SVN, there's no way to list unregistered
files.  A correct (but horribly slow) default implementation should also
be doable.

Bye,
Tassilo