bug#61817: 30.0.50; Project.el finds incorrect project roots in git worktrees

[Arthur Miller]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 27/02/2023 16:15, Arthur Miller wrote:
>
>>> We could change project-try-vc to follow the link to the parent repo, but 
>>> how is
>>> the rest of it going to work?
>>>
>>> If the project root is the parent repo, which set of files would 
>>> (project-files
>>> pr) return? And how could that be implemented?
>> I don't know how it works now, so I can't really tell you, but you have 
>> called
>> git root "actual git root", so perhaps something that differentiate between
>> directory hierarchy root (worktree), and real git root (project). Perhaps
>> project-vc-root, if you don't already use that name, or something similar?
>
> Do you mean you want a separate helper to get to the "worktree parent"
> repository?

I wouldn't call it "worktree parent", because it relatively git specific; if you
prefer to be project neutral; but something like that yes.

> repository? We could add something like that to VC, I guess. But your own code
> seems to be working well too.

I didn't meant to ask for someone to write it for me :), I can use my own
code, but as said I had somewhat different expectation from project.el when I
did this request. You can just close it, it is ok; I am sorry for the trouble.

> I haven't had a use for "worktree parent" myself, though. And until we get a
> good idea for how people will use a feature, it's a little difficult to 
> choose a
> place for it.

If you have a branch "main", work in a branch X, and would like to test/fix
something unrelated to X, wouldn't you create a new workree Y from clean "main"?
It is a bit special case, which I personally use, so to automate this, 
"worktree"
as root is not enough, I need the actual root. For other purposes I need the
worktree. So as said, both are needed. Of course, worktree will be used more
frequently for natural reasons.

More general case is to create patch for some unrelated branch Z, in which case
I am not sure, but I believe I need to exec git branch and/or git worktree list
and parse the output to see where to jump, and from which branch and directory
to create a new worktree.

>>>> Yes that is what I wanted? :-).
>>>> For automation purpose I need to find the project root, so I can pull 
>>>> sources
>>>> to
>>>> main, create a clean worktree from main, and switch Emacs to the new 
>>>> worktree
>>>> interactively in one command, like M-x make-new-patch. Emacs asks me for a 
>>>> name
>>>> and create a clean worktree from the main trunk for current project. 
>>>> Actually
>>>> better variant is to ask which branch to patch, but the first one is 
>>>> slightly
>>>> faster and works just fine in many cases.
>>>
>>> It sounds like your code is Git-specific, not project-neutral.
>> Yepp. Question is what project.el then really is, if it can't handle vc
>> specific
>> "projects", if it is only synomim for operations on directory trees? Perhpas 
>> it
>> should be called dirtree.el, because that is what it does: it works on
>> directory trees and uses "project markers" to know where to start/stop? I am 
>> not
>> trying to by sarcastic or negative.
>
> It uses VCS-specific information and converts it to a shape that should be
> usable without the caller being aware of which underlying data that 
> information
> is determined by.
>
> In particular, it reads .gitmodules, .git and .gitignore.
>
> And, well, different backends can make different decisions on the backends's
> behavior. But, ideally, it should be good for the majority of uses. Such as 
> our
> built-in commands, for example: project-vc-dir, project-eshell, etc.

Yes, they are nice, and they illustrate that for some purpose, worktree is
needed.

>> I don't mean it is not useful, we need both,
>> operations that work as projectile/project, on directory trees, but also 
>> those
>> that actually understand a possible project structure. To me projectile and 
>> now
>> project.el seems more like they are an extension to Dired, not in some 
>> negative
>> connotation, but at least conceptually, then really project handling. I am 
>> not
>> trying to be negative, I am a bit oversimplistic and exaggerating, becuase I 
>> am
>> trying to illustrate the difference how I think about "projects".
>
> It a way -- yes, because we describe a project as a set of files. With a
> well-defined root. But it doesn't absolutely have to be in the shape of a 
> single
> directory tree, though it usually is.
>
> Whenever one chooses a different shape, though, they should consider how that
> will affect the common uses.

It is not about the shape; you can put a worktree as a subdirectory in a main
repository, it does not need to be out of source tree; project.el will still not
find the real git root. You can test it yourself in Emacs source:

In Emacs repo create new worktree: git worktree add new-test; change to newly
created folder and test your tools, they will work only on new-test.

>> Version control has become an integral part of all projects, at least in
>> software industry, just as building projects, boilerplate generating projects
>> etc. What I am saying is that project-neutral is OK, we don't want just a
>> different name for Git commands, but any project library nowadays should be 
>> able
>> to work with vc systems, build systems etc. At least for software management
>> projects.
>
> You seem to be assigning a lot of meaning to the fact that the root doesn't
> point to "worktree root", but to the current repository root (modulo the 
> search
> in the parent directory when the current repo is detemined to be a submodule).
>
> But would it be a better choice?

I don't think it is either one; definitely not an exclusive choice. I tried to
say that we need both.

But the root :) of this discussion, is that I missunderstood project.el as an
attempt to create a useful middleware for general working with projects, but I
guess you and other authors are more seeing it as an end-user application. I
appologize, I should have probably looked more into the code and asked. I just
tried to use some functions :). 

> When I use worktrees, it's to carry on development in a certain branch. 
> Meaning,
> I need to be able to pull/push/commit in that branch, all within the same
> worktree.

Yes, in some cases that is a workflow. But there are cases when its prefered to
checkout other branch, while work on something else; and as said above, in that
case is prefered to make a new worktree from either "clean" main or some other 
branch.

> To do a pull, or to run some commands in that worktree, I can call
> project-eshell. To do a commit, I can call up project-vc-dir. In both cases, 
> the
> useful thing to do for 'project-root' is to return the current repository 
> root,
> not the worktree parent. Because you don't make new commits in the parent repo
> when working on a worktree, you make them in the current one.

In this cases yes; but in some other cases no. Uses you mention are definitely
more frequent, then checking out unrelated branch, but I don't think it is so
uncommon either.

> I'm all for adding new Version Control related features, but they should be
> based off realistic, specific user scenarios.

Automation. I don't want to open terminal (or magit, does not matter), switch
branches and do that stuff manually (if I don't have to).

>>> But you still could find the worktree root using project.el, and then read 
>>> the
>>> contents of .git, follow the link and do your automation stuff.
>> Yeah, but it what does it saves me? In Git case it is really synonym to:
>>      (git-dir (locate-dominating-file directory ".git"))
>> I can just as well call built-in function instead of requiring project.el.
>
> Sure. locate-dominating-file is an underrated function. If you just want to 
> find
> the current repository root (no matter if it's a submodule or etc), the above
> will easily work. project-try-vc adds some caching on top of that, but you 
> might
> not need it.

I didn't know you have caching in project.el; that is a good feature to have. I
personally don't need it, but definitely good to know it is there.

>> As said I thought project.el was more general to work with projects on a 
>> deeper
>> level, sort-of EDE replacement or complement, and I wanted to use it as 
>> library,
>> but I understand now it is not.
>
> The idea was for a more simple/transparent and extensible EDE alternative. Or 
> an
> extension point to plug Projectile in.

Ede is extensible; it is just a bit arcane and there is not too much
documentation, unfortunately. I would prefer if Ede was re-used instead of the
duplicated effort, but I do understand if you want to replace it.

> It can still grow some additions (such as build tool support), but for now we
> have what we have.

In my opinion, Emacs lacks

1) a good and easy to use boiler-plate generator framework. Ede is meant for 
that,
but it is arcane to use and is meant to be used with existing projects. Also, 
Emacs
will automatically create a new folder if asked to find-file, when folder does
not exists. But, if a foder exists, it creates new file in that folder, which 
is not
preferred behavior.

In the end, as an end-user I have to manually create folders or run scripts
before I can ask Ede to fill-in with some code generator.

There is no reason to not automate project generation. We do have all the tools
in Emacs already but I haven't seen any good generator. Maybe there is some, but
I am not aware of one.

I am playing with a little framework based on org-capture; but it is nothing
polished I wish to share with anyone. With boilerplate I mean init git, generate
readme, lsp database, makefiles and that sort of stuff. I think a combo between
Ede and project.el would be nice.

2) better integration with tools for common workflow(s). Adding
features, fixes, tests etc, is mostly about creating a new patch, which with
addition of worktrees in Git can be super nicely automated. For example: add new
library, or add new executable, or add new fix, it all boiles down to add new
branch which leads to add new worktree, add a possible target in Makefile,
create possibly a test file, probably some other thing. It would be nice to be
able to say: M-x add-new-shared-library or M-x add-new-executable, or M-x
add-new-fix and Emacs will do at least initial boilerplate.

If you plan to add build tool support to project.el, it would be welcome :).