Re: policy discussion on bundling ELPA packages in the emacs tarball - take 2

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Fri, 29 Jan 2021 11:32:34 -0800
> 
> Attached is an updated version of emacs/admin/notes/elpa, describing a
> proposal for bundling ELPA packages in the emacs distribution.
> 
> I believe I have included all the recent discussions.

Thanks.  The text confused me a couple of times, so I think it "needs
some work"; see the questions and comments below.

> I think I figured out a way to resolve the submodles vs worktrees
> dilemma; each emacs developer can choose. If they don't execute any
> submodule commands, they can use worktrees.

I think the consequences of each of the two choices shall be spelled
out, because at least to me they are not clear.  What are the
advantages and disadvantages of each choice?

> If we can reach consensus on this, the next step would be to create a
> feature branch and edit the code to implement this, then do a practice
> release with that branch.

I'm not sure we need a public branch for that.  What would be the
purpose of such a branch?

> After the initial install of Emacs, users do not need to do anything
> special to update a bundled ELPA package; the normal package.el
> mechanisms recognize all the versions properly.

This should tell more about "not having to do anything special": I'm
guessing (or rather hoping) you allude here to upgrading/downgrading
the bundled packages from ELPA, *not* to how the bundled packages are
used without upgrading.  IOW, to just use the version shipped with the
Emacs release tarball, users don't need to use package.el, nor will
they need to add any package-related code into their init files or
create early-init.el if they didn't have it before.

That is, what does this mean from the POV of a user who didn't until
now use package.el at all, but just used what came with the release
tarball?

> Emacs core code may not depend on (bundled or non-bundled) GNU ELPA
> packages unless they are ":core" packages; if that functionality is
> required, the package must be moved to Emacs.

Another unclear part.  The following questions arise, and I don't see
any answers to them:

  . how to know whether a package is ":core", and how to indicate that
    it's ":core"? is the elpa-packages file the only place?

  . "if that functionality is required, the package must be moved to
    Emacs" -- what functionality is alluded to here, and what does it
    mean "moved to Emacs"? does any part of the following description
    apply to such "moved to Emacs" packages?

>    - A simple git clone/checkout of emacs.git master creates a
>      workspace will all code required to build and run Emacs.

What does this mean in practical terms?  Is this any change from the
current situation, and if so, how?  And what is "workspace" in this
context?

>    - Any ELPA package can be changed without affecting Emacs core.

How is this related?  What changes to ELPA packages and in what
location (on Savannah, on my local clone of elpa.git, in my clone of
emacs.git, something else?) will not "affect Emacs core", and what
does "affect" mean in this context?

> For single file packages, the file can simply be in both elpa.git and
> emacs.git master. The package is marked ":core" in the ELPA
> "elpa-packages" file. For example, eldoc and flymake are :core
> packages.

So to know whether a package is ":core", one must look in the local
clone of elpa.git?  What if there's no local clone of elpa.git: does
it mean Emacs (and package.el specifically) won't know whether a
package is ":core"? does that matter?

> This mechanism is not supported for multi-file packages, because
> maintaining sync between the files in emacs.git and elpa.git is more
> problematic.
> 
> To bundle a multi-file package in Emacs, add to elpa.git a branch
> `emacs-NN/[PKGNAME]` which holds the version to be included in the
> next Emacs release of version NN:
> 
>     git branch -c externals/[PKGNAME] emacs-NN/[PKGNAME]
> 
> Note that every GNU ELPA package has a branch `externals/[PKGNAME]` in
> elpa.git.

My Git documentation doesn't mention the -c switch of "git branch".
What did I miss?

More importantly, why do we need more than one protocol for bundling a
package?  Wouldn't it be simpler to have just one?

> Also push the release branch to emacs.git:
> 
>     git push [USERNAME]@git.savannah.gnu.org:/srv/git/emacs.git 
> emacs-NN/[PKGNAME]
> 
>     This allows emacs developers to checkout the bundled package
>     without requiring a separate clone of elpa.git.
>     
> In emacs/master, add a git submodule for that branch:
> 
>     git submodule add --reference . -b emacs-NN/[PKGNAME] elpa/[PKGNAME]

You say "emacs/master", and I'm guessing this means the master branch
of emacs.git (but please be explicit about this, and don't let us
guess).  If that is correct, then the master branch is not the most
important place to do this: the most important place is the emacs-NN
release branch, because that's where we care the most about having the
stable version of each package.

And finally, what does the above "git push" command do?  You are
pushing from one Git repository (elpa.git) to another (emacs.git),
right?  What will that do in emacs.git, and what would be the effect
on users who will next "git pull" from emacs.git?

> Emacs developers should add '--recurse-submodules' to the 'git clone'
> command when cloning Emacs; that will checkout the submodules with the
> rest of Emacs. To checkout the submodules in an existing Emacs
> directory tree:
> 
>     git submodule update --reference . --init

To which branch is this applicable?  E.g., I have a separate clone
where I work on the current release branch (emacs-27 as of this
writing); do I need to do the above in that clone as well, and should
the command change to tell Git that I want the versions of the bundled
packages that are appropriate for the branch I'm tracking there?  What
if I switch the branch inside a repository -- do I need to do anything
special for the submodules?

>   We provide a script `checkout_git_elpa_worktrees.sh' that uses 'git
>   worktree' to checkout each bundled ELPA branch into emac/elpa,
>   instead of 'git submodule update'.

What does this script do? does it allow developers to use "git
worktrees" safely without losing the access to bundled packages?  IOW,
more details about the script need to be spelled out.

> There are currently (jan 2020) some multi-file packages with code in
> both elpa.git and emacs.git; org, gnus for example. To convert those
> packages into bundled packages, they are deleted from emacs.git, and
> then follow the process above. The net result in the emacs directory
> tree is that the code moves from emacs/lisp/[PKGNAME] to
> emacs/elpa/[PKGNAME].

How will this move affect Git history and the related "git log" and
"git blame" commands? what about "git bisect" across the move?

> At Emacs startup, the (installed or source) `emacs/elpa' directory is
> added to `package-directory-list', and is treated the same as other
> entries in that list. Note that this is different from files in
> emacs/lisp; those are added to `load-path'.

What does this mean from the POV of autoloading?  E.g., when I run the
latest Emacs from the current master branch, package-directory-list is
nil; does this mean Emacs will be unable to load packages in
emacs/elpa without my setting package-directory-list to something
non-nil?  Or will package-directory-list be set up automatically at
startup?  If the latter what will be the order of searching packages:
first load-path, then package-directory-list, or vice versa?

> Using `package-directory-list' allows users to disable bundled
> packages via `package-load-list'.

What does "disable" mean in this context?  Why would users want to
"disable" a bundled package?

> - One issue is whether the autoloads of bundled ELPA packages are
>   processed when we dump Emacs (like we do for all the packages that
>   are in Emacs core), or whether that's done during
>   `package-activate-all`. The baseline design does it during
>   `package-activate-all'.
> 
>   Doing it at dump time gives better startup times, at the cost of
>   making it impossible for the end-user to prevent activation of a
>   package.

That goes back to the question why would users want to prevent
activation of a bundled package.

>   Users who care about startup time can do their own dump.

If this is an acceptable solution (given that re-dumping is still not
reliable enough and has known bugs), why not turn the table and say
that users who want to prevent activation of a package can do their
own dump?

> - emacs/elpa/[PKNAME] vs emacs/lisp/[PKGNAME]
> 
>   The baseline design keeps all bundled ELPA packages in
>   emacs/elpa/[PKNAME], both in the source and installed directory
>   trees. This makes it very easy to distinguish an emacs directory
>   tree with bundled packages from one without.
> 
>   An alternative is to keep the bundled ELPA packages in
>   emacs/lisp/[PKGNAME]; that minimizes the change when a package that
>   is current in emacs.git converts to a bundled package. This would
>   also mean that bundled packages are put in `load-path' at Emacs
>   startup, not in `package-directory-list'.

I prefer the latter, unless it's really impractical.

> - Define a useful emacs lisp project (or something simpler),
>   So `C-x p f' searches `load-path' via `project-find-file'
> 
>   That reduces the need to remember what packages are bundled, and
>   where you have them checked out.

I couldn't follow this item.  Can you elaborate?

> - Automating compiling C, Ada, Rust or other language code required by
>   the package, to executables or modules. This includes defining
>   standard places to put executables, modules, and other
>   aritifacts. This is more important for bundled packages; users
>   expect Emacs to "just work" out of the box.

I'm not sure I understand the issue: the standard places for all of
this are already defined.  Why would they need to change under this
proposal?