Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Fri, 29 Jan 2021 11:32:34 -0800
>> 
>> Attached is an updated version of emacs/admin/notes/elpa, describing a
>> proposal for bundling ELPA packages in the emacs distribution.
>> 
>> I believe I have included all the recent discussions.
>
> Thanks.  The text confused me a couple of times, so I think it "needs
> some work"; see the questions and comments below.

Thanks for the detailed review. 

>> I think I figured out a way to resolve the submodles vs worktrees
>> dilemma; each emacs developer can choose. If they don't execute any
>> submodule commands, they can use worktrees.
>
> I think the consequences of each of the two choices shall be spelled
> out, because at least to me they are not clear.  What are the
> advantages and disadvantages of each choice?

With submodules:

    - a single command 'git clone <emacs> --recurse-submodules'
    populates the entire Emacs directory tree.

    - vc understands submodules
    
    - cannot use any git worktrees.
    
With worktrees:

    - developer must remember to not execute any submodule commands

    - a second command ./checkout_git_elpa_worktrees.sh must be used to
    finish populating the Emacs directory tree

    - developer can use git worktrees to maintain multiple checkouts
    from the local emacs.git repository
    
I'll add this to the text.

>> If we can reach consensus on this, the next step would be to create a
>> feature branch and edit the code to implement this, then do a practice
>> release with that branch.
>
> I'm not sure we need a public branch for that.  What would be the
> purpose of such a branch?

I'd like Phillip Lord to test packaging windows, at least. But I'll
start with a private branch.

>> After the initial install of Emacs, users do not need to do anything
>> special to update a bundled ELPA package; the normal package.el
>> mechanisms recognize all the versions properly.
>
> This should tell more about "not having to do anything special": I'm
> guessing (or rather hoping) you allude here to upgrading/downgrading
> the bundled packages from ELPA, *not* to how the bundled packages are
> used without upgrading.  IOW, to just use the version shipped with the
> Emacs release tarball, users don't need to use package.el, nor will
> they need to add any package-related code into their init files or
> create early-init.el if they didn't have it before.

Perhaps it would be best to say:

    For users, bundled packages are treated the same as any other
package, except that they are pre-installed.

> That is, what does this mean from the POV of a user who didn't until
> now use package.el at all, but just used what came with the release
> tarball?

They don't have to do anything.

>> Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
>> packages unless they are ":core" packages; if that functionality is
>> required, the package must be moved to Emacs.
>
> Another unclear part.  The following questions arise, and I don't see
> any answers to them:
>
>   . how to know whether a package is ":core", and how to indicate that
>     it's ":core"? is the elpa-packages file the only place?

Yes, but it actually doesn't matter. A better way to say this is:

    Emacs core code may not depend on any code in the emacs/elpa
directory.

>   . "if that functionality is required, the package must be moved to
>     Emacs" -- what functionality is alluded to here, 

the functionality of some package.

>   and what does it mean "moved to Emacs"? 
>   does any part of the following description apply to such "moved to
>   Emacs" packages?

I don't describe that process. I'll add it.

>>    - A simple git clone/checkout of emacs.git master creates a
>>      workspace will all code required to build and run Emacs.
>
> What does this mean in practical terms?  

The command 'git clone <emacs.gi> --recurse-submodules' creates an Emacs
directory tree.

It's probably best to just delete this paragraph; that command is given
later.

> Is this any change from the current situation, and if so, how? And
> what is "workspace" in this context?

The only change is the addition of --recurse-submodules

By "workspace" I mean "emacs diretory tree".

>>    - Any ELPA package can be changed without affecting Emacs core.
>
> How is this related?  

It's a consequence of the rule "Emacs core code cannot depend on ELPA packages"

> What changes to ELPA packages and in what location (on Savannah, on my
> local clone of elpa.git, in my clone of emacs.git, something else?)

Any; that's the point.

> will not "affect Emacs core", and what does "affect" mean in this
> context?

The normal English meaning; no byte compile errors, no change in behavior.

>> For single file packages, the file can simply be in both elpa.git and
>> emacs.git master. 

Stefan Monnier: I think this is incorrect; there is no copy in elpa.git?
For example, 'cl-print' is a :core package, but there is no
externals/cl-print branch, and no file 'cl-print' in the master branch.
It appears that elpaa--core-package-sync links the file to a checkout
(on the ELPA web server) of the emacs.git repository. And that it can
handle directories.

>> The package is marked ":core" in the ELPA >> "elpa-packages" file.
>> For example, eldoc and flymake are :core >> packages.
>
> So to know whether a package is ":core", one must look in the local
> clone of elpa.git?  

Yes.

> What if there's no local clone of elpa.git: does it mean Emacs (and
> package.el specifically) won't know whether a package is ":core"? 

Yes.

> does that matter?

No.

The Emacs directory tree contains a copy of the package file (in
emacs/lisp/...). The ELPA admin code makes that available as a package
that can be installed via package.el. The package maintainer can release
a new version of the package by changing the version number and
committing to emacs.git.

>> This mechanism is not supported for multi-file packages, because
>> maintaining sync between the files in emacs.git and elpa.git is more
>> problematic.

This should say "sync between the files in emacs.git and upstream".

>> To bundle a multi-file package in Emacs, add to elpa.git a branch
>> `emacs-NN/[PKGNAME]` which holds the version to be included in the
>> next Emacs release of version NN:
>> 
>>     git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME]
>> 
>> Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in
>> elpa.git.
>
> My Git documentation doesn't mention the -c switch of "git branch".
> What did I miss?

https://git-scm.com/docs/git-branch

git branch (-c | -C) [<oldbranch>] <newbranch>

I may have the wrong option; that page does not say "create a new
branch from an existing branch", but I think that's what -c does.

> More importantly, why do we need more than one protocol for bundling a
> package?  Wouldn't it be simpler to have just one?

I guess you mean :core is one protocol, and create emacs-NN/[PKGNAME]
branch is another? I was not using the term "bundled" to refer to :core,
but you are correct that they are similar.

The main rationale for having :core is that it requires no change in
emacs.git to make a package available via ELPA.

The main rationale for having emacs-NN/[PKGNAME] is that it allows using
'git push/pull' to sync with an upstream development repository.

:core requires manual sync with an upstream repository; the developer
must use ediff or equivalent, not 'git push/pull'. 

>> Also push the release branch to emacs.git:
>> 
>>     git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git 
>> emacs-NN/[PKGNAME]
>> 
>>     This allows emacs developers to checkout the bundled package
>>     without requiring a separate clone of elpa.git.
>>     
>> In emacs/master, add a git submodule for that branch:
>> 
>>     git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME]
>
> You say "emacs/master", and I'm guessing this means the master branch
> of emacs.git (but please be explicit about this, and don't let us
> guess).  

Yes.

> If that is correct, then the master branch is not the most important
> place to do this: the most important place is the emacs-NN release
> branch, because that's where we care the most about having the stable
> version of each package.

Yes. 

> And finally, what does the above "git push" command do?  You are
> pushing from one Git repository (elpa.git) to another (emacs.git),
> right?  

Right. It copies all required files and related meta info from the
elpa.git repository to the emacs.git repository.

> What will that do in emacs.git, 

Similar to any other branch push; the branch is available for checkout.

> and what would be the effect on users who will next "git pull" from
> emacs.git?

None, until they checkout a local copy of the new branch, either
directly or via 'git submodule update'.

>> Emacs developers should add '--recurse-submodules' to the 'git clone'
>> command when cloning Emacs; that will checkout the submodules with the
>> rest of Emacs. To checkout the submodules in an existing Emacs
>> directory tree:
>> 
>>     git submodule update --reference . --init
>
> To which branch is this applicable?  

master or emacs-nn where nn >= 28.

> E.g., I have a separate clone where I work on the current release
> branch (emacs-27 as of this writing); do I need to do the above in
> that clone as well, 

No, because we are only proposing to introduce this feature in emacs 28
(or later).

> and should the command change to tell Git that I want the versions of
> the bundled packages that are appropriate for the branch I'm tracking
> there? 

Assuming we have reached the point where emacs > 28 are available, the
file .gitmodules contains the submodule info, including the branch name;
'git submodule update' uses that file to determine what to do.

.gitmodules is committed to emacs.git.

I think there may be an issue here; if .gitmodules records the branch
name for a submodule, but not a specific commit, and there are multiple
commits on that submodule branch, then attempting to checkout anything
other than HEAD from emacs-nn might get the wrong version of the
submodule. That could be fixed by manually checking out the correct
version in the submodule directory.

> What if I switch the branch inside a repository -- do I need to do
> anything special for the submodules?

You need to use 'git checkout --recurse-submodules'; that updates the
submodules properly.

>>   We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git
>>   worktree' to checkout each bundled ELPA branch into emac/elpa,
>>   instead of 'git submodule update'.
>
> What does this script do? 

For each bundled package:

    git worktree add -b emacs-NN/[PKGNAME] elpa/[PKGNAME] origin/[PKGNAME]

although the details may vary depending on whether it's been done before
or not; git gets confusing sometimes.

> does it allow developers to use "git worktrees" safely without losing
> the access to bundled packages? 

Yes.

>> There are currently (jan 2020) some multi-file packages with code in
>> both elpa.git and emacs.git; org, gnus for example. To convert those
>> packages into bundled packages, they are deleted from emacs.git, and
>> then follow the process above. The net result in the emacs directory
>> tree is that the code moves from emacs/lisp/[PKGNAME] to
>> emacs/elpa/[PKGNAME].
>
> How will this move affect Git history and the related "git log" and
> "git blame" commands? what about "git bisect" across the move?

There is already separate history in emacs.git and elpa.git. The history
in emacs.git will simply end; the history in elpa.git will simply
continue.

>> At Emacs startup, the (installed or source) `emacs/elpa' directory is
>> added to `package-directory-list', and is treated the same as other
>> entries in that list. Note that this is different from files in
>> emacs/lisp; those are added to `load-path'.
>
> What does this mean from the POV of autoloading?  E.g., when I run the
> latest Emacs from the current master branch, package-directory-list is
> nil; does this mean Emacs will be unable to load packages in
> emacs/elpa without my setting package-directory-list to something
> non-nil?  

The startup code will add emacs/elpa to `package-directory-list', and
`package-activate' will load the autoloads file, so you don't have to do
anything.

> Or will package-directory-list be set up automatically at startup? If
> the latter what will be the order of searching packages: first
> load-path, then package-directory-list, or vice versa?

package-directory-list is not used to find files for 'load'; it is used
to find packages for `package-activate'. When `package-activate' runs,
it prepends the activated package directories to `load-path'.

>> Using `package-directory-list' allows users to disable bundled
>> packages via `package-load-list'.
>
> What does "disable" mean in this context?  Why would users want to
> "disable" a bundled package?

I'm quoting the doc string for package-load-list; if `package-load-list'
contains ("foo" nil), then package "foo" is not available; it is
"disabled".

One reason to do this is if you are a package developer, and want to use
the development version of a package, not the installed version (I do
that for ada-mode).

Or there's something in the auto-load for an installed package that
interferes with something you are doing; perhaps you are working on an
alternative package for a similar feature.

For non-bundled package an alternative to "disable" is to uninstall the
package, but that's not an option with bundled packages.

The point is that package.el provides this mechanism to manage installed
packages; to be consistent, it should be available for bundled (ie
"pre-installed") packages as well.

>> - Define a useful emacs lisp project (or something simpler),
>>   So `C-x p f' searches `load-path' via `project-find-file'
>> 
>>   That reduces the need to remember what packages are bundled, and
>>   where you have them checked out.
>
> I couldn't follow this item.  Can you elaborate?

If you use C-x f (find-file), you need to know what top-level directory
to look in to find a package file; for example ada-mode.el will be in
emacs/elpa/ada-mode, not emacs/lisp/progmodes/ada-mode

If you use C-x p f (project-find-file), with an appropriate elisp
project active, then you have file-name completion on all elisp files in
load-path, so you don't have to know what directory to look in. Directory
information is only needed to disambiguate files with identical names
under emacs/lisp/cedet.

>> - Automating compiling C, Ada, Rust or other language code required by
>>   the package, to executables or modules. This includes defining
>>   standard places to put executables, modules, and other
>>   aritifacts. This is more important for bundled packages; users
>>   expect Emacs to "just work" out of the box.
>
> I'm not sure I understand the issue: the standard places for all of
> this are already defined.  

Ok, I was not aware of that.

-- 
-- Stephe