Proposals for fixing package-initialize

[Radon Rosborough]

Hello all,

You may have seen slightly over 150 emails fly by in the last few
weeks talking about package-initialize. Hilariously, we still haven't
managed to reach a conclusion.

This email is an attempt to present all the proposals that have been
discussed, in an even-handed way, so that we can make a decision
without everyone needing to have read the aforementioned 150 emails.

I will *not* be providing any context on the issue; see [1] for that,
but note that the discussion branched out so much that everyone's
positions are now completely different :P

==> Proposal A: Leave it alone

Summary:
  - call package-initialize in startup.el after loading the init-file
  - when package-initialize is called, also insert a call to it into
    the init-file, if one was not found there
  - this is the current behavior

Advantages:
  - requires no work
  - people are used to the current behavior
  - backwards compatible

Disadvantages:
  - modifying the user's init-file is ugly
  - doing it automatically is annoying and possibly dangerous
  - if the user has customized package.el, then Emacs will put
    package-initialize in the wrong place
  - this type of behavior is not common practice

==> Proposal B: Add a second init-file

Summary:
  - add a second init-file, ~/.emacs.d/early-init.el
  - this file is sourced earlier than init.el, during startup.el
  - move the call to package-initialize in startup.el from its current
    location to between early-init.el and startup.el
  - don't have package.el modify the init-file ever

Advantages:
  - does not require modifying the user's init-file
  - package.el still works out of the box
  - the only people affected by the change are the ones who have
    customized package.el
  - foolproof, never does the wrong thing (except for advanced
    existing configs that customize package.el)
  - people can also use early-init.el to disable things like the tool
    bar before they are loaded in the first place: two birds with one
    stone
  - the only people who need to know about early-init.el are advanced
    users who want to customize package.el
  - Customize works both for customizing package.el and also for
    customizing packages
  - does not require package-initialize to be put in the user's
    init-file, not even manually
  - makes it easy to disable package.el permanently

Disadvantages:
  - adds another init-file, which is additional complexity
  - we must add a mechanism to Customize that allows to specify the
    file into which a particular variable should be saved, so
    package.el can still be customized

==> Proposal B': Add an early-init special form

Summary:
  - same as above, but instead of a separate file, allow init.el to
    start with a special (with-early-config ...) form
  - if with-early-config form is found in init-file, load it when
    early-init.el would be loaded

Advantages:
  - we still only have one init-file

Disadvantages:
  - the startup sequence is a little weird
  - it's not clear how this would work with Customize

==> Proposal C: Add a template init-file

Summary:
  - if Emacs is started without an init-file, generate one from a
    template
  - the template init-file has a call to package-initialize in it, and
    some comments explaining where to put package and package.el
    configuration
  - the template init-file is not generated in 'emacs -Q' mode
  - get rid of package.el modifying the init-file

Advantages:
  - does not require modifying the user's init-file
  - package.el still works out of the box for new users
  - we could use the template init-file for encouraging users towards
    better defaults without breaking backwards compatibility

Disadvantages:
  - having Emacs automatically create files is not ideal
  - if we decide against using the template init-file for better
    defaults, then it's kind of awkward to just use it for package.el
  - doesn't work for users with existing init-files
  - still reqiures a call to package-initialize in the user's
    init-file

==> Proposal D: Hook into error handlers

Summary:
  - call package-initialize whenever a void-function or require error
    is encountered
  - get rid of package.el modifying the init-file
  - possibly get rid of call to package-initialize in startup.el?

Advantages:
  - doesn't require modifying the user's init-file
  - probably package.el still works out of the box

Disadvantages:
  - implications (performance, correctness, implementation details)
    are completely unexplored
  - not very elegant
  - business logic from the package manager is intermixed with a
    fundamental, generic part of the Elisp interpreter
  - hard to tell when package-initialize gets called, so best practice
    would probably still be to have package-initialize in the
    init-file
  - the fact that a core Lisp error causes lots of magic to happen
    will make debugging more "fun"

==> Proposal E: Call package-initialize multiple times

Summary:
  - add a second call to package-initialize before init.el is loaded
  - if package-initialize is called again, and package-load-list or
    other variables have been customized since the last time it was
    called, unload any packages that were previously loaded that now
    should not be loaded

Advantages:
  - does not require modifying the user's init-file
  - backwards compatible, at least if it works

Disadvantages:
  - impossible to disable package.el without introducing a magic
    dotfile (~/.emacs.d/.nopackage) or similar
  - impossible to unload packages correctly, in general, so this
    approach would be buggy in many edge cases
  - severe performance regressions if many packages are disabled using
    package-load-list
  - "As I said, there's no doubt that do+undo is a poor way to do
    nothing." "Again, here we're back to the obvious observation that
    do+undo is a poor and complicated way to do nothing." "Can't
    disagree that "do+undo" is a poor way to do nothing." -- Stefan
  - requires extra work to make multiple calls to package-initialize
    faster, not to mention figuring out how unloading is supposed to
    work, or if that's even possible in principle

==> Proposal F: Use more declarative configuration

Summary:
  - make it so that packages can be configured in a way that causes
    package-initialize to be called automatically

Advantages:
  - would solve all the problems

Disadvantages:
  - nobody has any idea how to do this
  - and it would probably take several years anyway

==> Proposal G: Parse the init-file

Summary:
  - call package-initialize in startup.el before loading init.el
  - parse the init-file in order to find out the values of
    package-load-list, etc. before running package-initialize

Advantages:
  - in the cases that it works, it would allow package.el to work out
    of the box without the need for package-initialize in the user's
    init-file
  - Emacs would not need to modify the user's init-file

Disadvantages:
  - would require solving the halting problem to work in general
  - is a horrifying hack

==> Proposal H: Make the user do it manually

Summary:
  - make it so that package.el doesn't modify the init-file, but don't
    change anything else
  - this is how it worked before the first "solution" to this problem
    was implemented

Advantages:
  - Emacs does not modify the user's init-file

Disadvantages:
  - package.el does not work out of the box, since package
    customizations will fail when put into the init-file
  - consequently we get lots of superfluous bug reports

==> My position

I think that the above advantages and disadvantages speak for
themselves, but to be clear, I think Proposal B is the best. It has
basically no practical disadvantages, solves every problem that is
tangled up in this issue with zero migration necessary except for very
advanced users, is trivial to implement, and simultaneously solves a
large class of other problems related to disabling things that are
enabled early during startup. Every other proposal has significant
practical disadvantages.

==> Postscript

I'd like to say thank you to Eli for pushing the question of whether
it would be practical to call package-initialize only in startup.el,
Timur for questioning why a second init-file would be so bad, and
Clément for explaining to me what was wrong with my original proposal.
Their arguments resulted in the discussion of what I now think is a
better proposal than my original one.

And yes, I really did review all 150 of those emails to gather
information for this summary :o

Best regards,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html