Re: Request to add Package to GNU ELPA

[Jonas Bernoulli]

Philip Kaludercic <philipk@posteo.net> writes:

> Jonas Bernoulli <jonas@bernoul.li> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Jonas Bernoulli <jonas@bernoul.li> writes:
>>>

>> Going one step further we could make package-vc available as a separate
>> package.  That would not be of much use *now*, but future improvements
>> would then be available to users of Emacs 29.
>
> I would in principle be interested in supporting this.

I am hoping that someone would take charge of Package on Emacs' side,
and bring some new momentum to its development.  You would be a good
candidate for that. ;D

Your package-vc is welcome addition (I haven't really tried it, since
my Borg serves me well) and Stefan certainly put a lot of work into his
elpa-admin, but Package itself has been a bit stagnant.

[I myself contribute to a lot of elisp packaging related projects, and
 do not plan to take on any new responsibilities; in fact I am trying
 to reduce them.]

>> I had a look at vc-clone and a few vc-*-clone.  They seem trivial
>> enough to backport, either as part of Compat or in package-vc.el.
>
> [...]

I assume that means "well it may *seem* that way, but if you are the one
actually doing the work...".  I know the feeling.  oO

>> [using a bleeding edge *ELPA]
>
> My experience was not that good, especially back before I got into Emacs
> development (or understood what was going on at all) I constantly ran
> into issues when updating packages from MELPA.  The result was that I'd
> usually just not update packages at all for long periods of time.  The
> next best thing for me was MELPA stable until NonGNU ELPA came around.

Using MELPA stable comes with its own problems.  MELPA's maintainers
advice users to stick to the bleeding-edge variant and use that
themselves.  And many users take that advice to heart, and many
developers are aware of that and act accordingly.

I don't think anyone is against moving towards relying more on released,
stable versions, but there seem to be some chicken and eggs involved.
We can all do a little bit to get closer to a more stable ecosystem, but
it will take time.

[I expect that once MELPA starts using sane version strings for snapshot
 packages, that will be a big step in that direction, because that makes
 it much simpler to properly specify the minimal required versions of
 dependencies, that work for both the bleeding-edge and stable channels.

 I am in fact working on that right now.  I was about to start rebuild
 all packages for testing purposes, but briefly got distracted by your
 reply to this. :P

 But there are other areas where improvements are needed, and I would
 like to leave it to others to tackle those.]

>>>> - I should also mention that my main motivation for pushing for this
>>>>   now, is that I would like to improve version handling.  That is a
>>>>   whole other can of worms, so I do not wish to discuss it just yet,
>>>>   to avoid distracting from the topic at hand, but I thought I should
>>>>   at least mention it.  You might very well end up being against my
>>>>   suggestions regarding versions strings, once I present them, but that
>>>>   should not be cause to oppose the change requested here as well.  Even
>>>>   if my suggestions regarding version strings are ultimately rejected, I
>>>>   still think it would be a good idea to add `package' to GNU ELPA, for
>>>>   the other reasons presented here.
>>>
>>> This implies that packages might themselves depend on newer versions of
>>> package.el -- which I think one should put some thought into before
>>> anything is done.
>>
>> How so? (Further down I suggest making some popular packages temporarily
>> depend on Package from GNU Elpa as a means to get that installed as
>> widely as possible, but that doesn't mean that those packages actually
>> *need* a newer Package.)
>>
>>> How can you e.g. change the way versions are handled
>>> in a way that people with older versions of package.el could still
>>> handle the change without confusion?
>>
>> You can't, and that is exactly the point I was trying to make.  Certain
>> things (including, but not limited to how version strings are handled)
>> are effectively set in stone, unless certain packages/features are made
>> available to users who, for whatever reason, stick to an old Emacs
>> release.
>
> Other than the version, it seems the other big one is the dependency
> list.  Is there anything else that could cause issues?

I can't think of any right now, but that of course doesn't mean we won't
discover any later on.

>>> The upgrading procedure for archive-contents has never appeared to me to
>>> be very robust.  Perhaps it would be better to introduce a second file
>>> (archive-contents.eld) with a more flexible format?
>>
>> An effort was made to provide an upgrade procedure -- the
>> "archive-contents" were prefixed with a version number.  But then
>> package.el was added to Emacs and we stopped to distribute it
>> separately.  Apparently we didn't realize at the time that this only
>> allowed us to tell the user "the archive and tool are not compatible,
>> deal with it".
>
> What could be done instead?  Should the updated version of package.el
> try to install a newer version of itself in case the version string is
> updated?

Yes, that's what I had in mind.

>> We could continue to distribute "archive-contents", which continues to
>> use version 1, and add "archive-contents.eld" which uses version 2.
>> But that would do nothing to get users to install a forward-compatible
>> version of Package (in the sense that once it becomes incompatible, it
>> will provide a smooth upgrade path to the new version).
>
> This seems like the better option to me.  Most people don't /need/ the
> newest version right away, and even if they did there would be no
> straightforward way of making sure all users would have it installed
> within some fixed time-frame.

We keep "archive-contents" going for as long as possible.  For the final
push to get users to upgrade, all packages except Package and its
dependencies could be removed from "archive-contents".  A pseudo package
could be added, whose summary (displayed in the list) and commentary
(displayed by describe-package) could be used to instruct users to
upgrade Package, and how.

>>   (We should invest some time to investigate how to make this as smooth
>>   as possible, but basically:)
>>
>> - 1. wget https://...package-2.0.0.tar
>>   2. tar -xf package-2.2.0.tar -C ~/.config/emacs/elpa/
>>   3. Restart Emacs and run package-refresh-contents again.
>
> This is exactly what I would like to avoid.

I am certainly interested in hearing better ideas.

> What happens when the .tar disappears and some user tries to fix this
> issue in a few years with these outdated instructions?

We should prevent that.  The tar file mentioned in the upgrade
instructions doesn't have to be hosted in the usual location only.
It could additionally be hosted somewhere else on gnu.org, where it
is safe from being discarded by elpa-admin or some cleanup script
that isn't aware of this special case.

> What if they use .emacs.d or some other directory?  For while there
> will be plenty of users that will figure this out, I know many others
> will be confused

Well, these steps were the executive summary, the instructions actually
shown to users should of course be more detailed and account for such
differences.

> (this also makes me think that a v2 should have support for some kind
> of MOTD system to explain issues like these).

+1

> So again, what would be the issue with an opt-in system to upgrading
> package.el, that could also be pushed forward by a few popular packages,
> without the need to break anything.

I am not against a long transitional phase, but I think that eventually
we will want to make some breaking change.

One example of an incompatible change is a change in how version strings
are handled.  As I said, I would like to advocate that in a separate
thread.  Here, it is just serves as an example, other incompatible
changes may become desirable.  Just the gist of it:

For a while [Non]GNU-devel ELPA used this version string format:

  VERSION.0-snapshotTIMESTAMP

"snapshot" has to be used because "archive-contents" contains the
version string in list format, e.g., (1 2 3 0 -4 20230405 111).  All of
"snapshot", "cvs", "git", "bzr", "svn", "hg", "darcs" and "unknown" are
encoded as -4 in the list format, but package-version-join maps it to
"snapshot".

That leads to very long version strings, which I assume is why Stefan
switched to just

  VERSION.0.TIMESTAMP

The ".0" serves as a pseudo separator.  Currently "1.0-snapshot" is
*smaller* than "1.0".  If we inject an additional ".0" then it is
smaller than "N.0", but not smaller than "N".

I would like to use a format that not only supported "pre-releases" but
also "post-releases".  Debian uses "post-releases" to implement
"debian-revision" [1]; we could use it to separate the "timestamp" part
from "latest released upstream version" part, in snapshot release
version strings.

[1]: https://manpages.debian.org/wheezy/dpkg-dev/deb-version.5.en.html

But again, maybe I am the only who feels a need to do something about
that; in this context it is just an example of a change that would break
backward compatibility; to demonstrate that some potential changes could
not be made merely opt-in, but instead would have to be either rolled
out to everyone or nobody.

     Jonas