Re: Unified project interface

[Dmitry Gutov]

On 07/31/2015 07:13 PM, Stephen Leake wrote:

> I often want to look at project.el (or xref.el, ada-mode.el, etc); I
> bind this function (currently using load-path) to M-F12, so M-F12
> proj <ret> finds it for me easily.

To reach the same result, I currently either use ido-switch-buffer (with 
ido-use-virtual-buffers set to t) or projectile-find-file (which asks 
the user to choose between files in the project, based on file names 
relative to the root). The key thing here is substring matching: I also 
simply type "project." and see lisp/progmodes/project.el as the only 
completion.

Using relative paths also helps with navigating to files with the same 
name inside different directories.

> Which is faster/friendlier depends on the details, so we'll have to wait
> for actual implementations. All I'm asking is that we do not make design
> decisions at the generic project API level that rule out this approach.

We do have to make some decisions, though. For instance, supporting both 
recursive and non-recursive elements in search-path will require a 
separate code path for each function that uses it.

As such, it will require a separate code path in xref-collect-matches 
and an alternative for xref--rgrep-command.

> > There are many directories inside /usr/include, but C_INCLUDE_PATH
> > won't mention most of them. Instead, #include <postgresql/pg_config>
> > will use /usr/include/postgresql/pg_config.h, as long as /usr/include
> > is in C_INCLUDE_PATH.
>
> Yes. So if the user wants to search all relevant includes for some
> declaration, they have to treat C_INCLUDE_PATH as recursive. But the
> compiler (which _defines_ C_INCLUDE_PATH) does _not_ treat it as
> recursive. So it's confused.

We need to search files in subdirectories as well, and that is the key 
distinction.

The fact that compiler treats the roots in a special way, however, can 
be important. Because of that, the list of roots contains information 
that can be lost after a naive translation to a list of "flat" 
directories. For instance, if you have the roots, you can search them 
linearly and tell whether there's a file corresponding to 
'postgresql/pg_config' inside one of them.

With a flat list, we could only give an imprecise answer to that 
question, and the search might have to perform extra work.

> I think cedet is the only exception to the 'flat load-path' rule (at
> least in core emacs); I hope it goes away, and is not copied by other
> projects.

Even though it's not very popular, it's unlikely to go away. But in any 
case, the corresponding path variables in all other ecosystems that I 
know of (Ruby, Python, Java, Node.js) follow in the footsteps of C and 
Emacs.

And the principle of translating qualified package (or module, or class) 
names to file names is quite pervasive.

Ada is an exception, rather than the rule.

> I would provide a special syntax to handle this in
> project-add-search-path:
>
> (defun project-add-search-path (project path &optional make-recursive)
>    "Add PATH (a list of directories) to the search path for PROJECT.
>
> If a directory in PATH ends in \"/**\", also add all directories under
> that directory.
>
> If MAKE-RECURSIVE is non-nil, the full project search path is pruned so
> that only highest-level directories are present, and all uses of
> `project-search-path' recurse into subdirectories. In this case, it is
> likely you will also need to specify ignored directories with
> `project-add-ignores'.

This description explains the purpose, but doesn't describe the 
resulting data structure that would hold that information. The ones that 
come to mind are rather ugly.

> If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse
> into subdirectories."
> ... )
>
> The Ada project files support the ** sytnax.

The /** syntax is rather nifty. But why do you propose to have it and 
MAKE-RECURSIVE at the same time? We could just interpret /** as the 
recursive marker. Then the result could be a list of strings.

> This could be handled if the user can explicitly specify project ignores
> (not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-*
> and ada directories). More complicated structures are harder to handle
> with ignores; you'll end up with a long flat list of ignores, so it
> might be simpler to have a flat list of includes instead.

I've just started to write about this... Yes indeed, I think every flat 
structure can be translated into path/to/dir, plus path/to/dir/*/ in the 
ignores. The translation function shouldn't take too much work to write.

There's no need to produce the list of ignores manually. The Ada project 
backend could do that for the user. And if the list is long... first, 
you can use wildcards. Second, at least xref-collect-matches won't have 
to handle the flat case specially. Nor will the other consumers of the 
project API.

> It would be surprising to an Ada user (used to flat search paths), if the
> Emacs project silently started using recursive search paths.

The Ada project implementation can provide a friendlier interface/set of 
customizations/etc for Ada users. Different communities are used to 
different things. Any default would be surprising to someone.

> > >       It is not at all clear how to convert a flat path into a recursive path
> > >       with ignores; the solution you have for elisp load-path is a kludge,
> > >       and could easily be wrong for some users.
> >
> > I'd like to see one of those users first.
>
> Me, and many other teams at NASA - structures similar to the above,
> where the non-included directories can also be release version,
> debug/release options, target hardware/os option, compiler
> vendor/version, etc.

There can be different opinions about this. As a project maintainer, I'd 
rather be able to find all references to the function foo, no matter the 
target OS of the code, or the targeted compiler version.

> The jdee project has lisp and java directories.

The Java class path is also "recursive".

> The kludge is in the root implementation of project-search-path, so it
> applies to all project backends that only customize
> project-search-path-function.

Please, stop calling major modes that do that "project backends". 
Customizing project-search-path-function does not constitute one.

A real backend can always override project-search-path (and smart ones 
should).

> > > So move it to "location" instead of "xref". The longest journey starts
> > > with one step ...
> >
> > Until it's used in at least two different places, creating a separate
> > package for it is, I think, premature.
>
> Right. We now have those two places; xref-find-definitions,
> project-find-regexp.

Both still use xref for display. The location values aren't consumed by 
anything else. So that's just one place.

> Although it would be even better if project-find-regexp uses the compile
> location data structure wrapped in a location-compile class, derived
> from the root location class, renamed from xref-location.

You're welcome to explore this direction with code.