Re: Request to add Package to GNU ELPA

[Philip Kaludercic]

(I apologise for the last message, accidentally sent it out before
writing anything (somehow?))

Jonas Bernoulli <jonas@bernoul.li> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Jonas Bernoulli <jonas@bernoul.li> writes:
>>
>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>
>>>> Jonas Bernoulli <jonas@bernoul.li> writes:
>>>>
>
>>> Going one step further we could make package-vc available as a separate
>>> package.  That would not be of much use *now*, but future improvements
>>> would then be available to users of Emacs 29.
>>
>> I would in principle be interested in supporting this.
>
> I am hoping that someone would take charge of Package on Emacs' side,
> and bring some new momentum to its development.  You would be a good
> candidate for that. ;D

I am glad to help where I can (e.g. making sure that package.el can be
distributed on ELPA), but I don't know exactly what you have in mind
with the other changes so I would rather let you tackle those issues.

> Your package-vc is welcome addition (I haven't really tried it, since
> my Borg serves me well) and Stefan certainly put a lot of work into his
> elpa-admin, but Package itself has been a bit stagnant.
>
> [I myself contribute to a lot of elisp packaging related projects, and
>  do not plan to take on any new responsibilities; in fact I am trying
>  to reduce them.]

>>> I had a look at vc-clone and a few vc-*-clone.  They seem trivial
>>> enough to backport, either as part of Compat or in package-vc.el.
>>
>> [...]
>
> I assume that means "well it may *seem* that way, but if you are the one
> actually doing the work...".  I know the feeling.  oO

I am not sure what you are talking about here?


[...]

>>>>> - I should also mention that my main motivation for pushing for this
>>>>>   now, is that I would like to improve version handling.  That is a
>>>>>   whole other can of worms, so I do not wish to discuss it just yet,
>>>>>   to avoid distracting from the topic at hand, but I thought I should
>>>>>   at least mention it.  You might very well end up being against my
>>>>>   suggestions regarding versions strings, once I present them, but that
>>>>>   should not be cause to oppose the change requested here as well.  Even
>>>>>   if my suggestions regarding version strings are ultimately rejected, I
>>>>>   still think it would be a good idea to add `package' to GNU ELPA, for
>>>>>   the other reasons presented here.
>>>>
>>>> This implies that packages might themselves depend on newer versions of
>>>> package.el -- which I think one should put some thought into before
>>>> anything is done.
>>>
>>> How so? (Further down I suggest making some popular packages temporarily
>>> depend on Package from GNU Elpa as a means to get that installed as
>>> widely as possible, but that doesn't mean that those packages actually
>>> *need* a newer Package.)
>>>
>>>> How can you e.g. change the way versions are handled
>>>> in a way that people with older versions of package.el could still
>>>> handle the change without confusion?
>>>
>>> You can't, and that is exactly the point I was trying to make.  Certain
>>> things (including, but not limited to how version strings are handled)
>>> are effectively set in stone, unless certain packages/features are made
>>> available to users who, for whatever reason, stick to an old Emacs
>>> release.
>>
>> Other than the version, it seems the other big one is the dependency
>> list.  Is there anything else that could cause issues?
>
> I can't think of any right now, but that of course doesn't mean we won't
> discover any later on.

Right, I was just thinking that it would make sense to have some list.

>>>> The upgrading procedure for archive-contents has never appeared to me to
>>>> be very robust.  Perhaps it would be better to introduce a second file
>>>> (archive-contents.eld) with a more flexible format?
>>>
>>> An effort was made to provide an upgrade procedure -- the
>>> "archive-contents" were prefixed with a version number.  But then
>>> package.el was added to Emacs and we stopped to distribute it
>>> separately.  Apparently we didn't realize at the time that this only
>>> allowed us to tell the user "the archive and tool are not compatible,
>>> deal with it".
>>
>> What could be done instead?  Should the updated version of package.el
>> try to install a newer version of itself in case the version string is
>> updated?
>
> Yes, that's what I had in mind.

Sounds good.

>>> We could continue to distribute "archive-contents", which continues to
>>> use version 1, and add "archive-contents.eld" which uses version 2.
>>> But that would do nothing to get users to install a forward-compatible
>>> version of Package (in the sense that once it becomes incompatible, it
>>> will provide a smooth upgrade path to the new version).
>>
>> This seems like the better option to me.  Most people don't /need/ the
>> newest version right away, and even if they did there would be no
>> straightforward way of making sure all users would have it installed
>> within some fixed time-frame.
>
> We keep "archive-contents" going for as long as possible.  For the final
> push to get users to upgrade, all packages except Package and its
> dependencies could be removed from "archive-contents".  A pseudo package
> could be added, whose summary (displayed in the list) and commentary
> (displayed by describe-package) could be used to instruct users to
> upgrade Package, and how.

OK, but this is something I don't expect to take place soon.

>>>   (We should invest some time to investigate how to make this as smooth
>>>   as possible, but basically:)
>>>
>>> - 1. wget https://...package-2.0.0.tar
>>>   2. tar -xf package-2.2.0.tar -C ~/.config/emacs/elpa/
>>>   3. Restart Emacs and run package-refresh-contents again.
>>
>> This is exactly what I would like to avoid.
>
> I am certainly interested in hearing better ideas.

For one thing an Elisp script would be preferable to avoid issues with
people who might not have wget or a specific version of tar installed.

But otherwise I would just try and avoid the need to force users to
upgrade.  Upgrading should be incentivized with the advantages of doing
so, not the issues of (perhaps unconsciously) holding back.

>> What happens when the .tar disappears and some user tries to fix this
>> issue in a few years with these outdated instructions?
>
> We should prevent that.  The tar file mentioned in the upgrade
> instructions doesn't have to be hosted in the usual location only.
> It could additionally be hosted somewhere else on gnu.org, where it
> is safe from being discarded by elpa-admin or some cleanup script
> that isn't aware of this special case.
>
>> What if they use .emacs.d or some other directory?  For while there
>> will be plenty of users that will figure this out, I know many others
>> will be confused
>
> Well, these steps were the executive summary, the instructions actually
> shown to users should of course be more detailed and account for such
> differences.
>
>> (this also makes me think that a v2 should have support for some kind
>> of MOTD system to explain issues like these).
>
> +1
>
>> So again, what would be the issue with an opt-in system to upgrading
>> package.el, that could also be pushed forward by a few popular packages,
>> without the need to break anything.
>
> I am not against a long transitional phase, but I think that eventually
> we will want to make some breaking change.
>
> One example of an incompatible change is a change in how version strings
> are handled.  As I said, I would like to advocate that in a separate
> thread.  Here, it is just serves as an example, other incompatible
> changes may become desirable.  Just the gist of it:
>
> For a while [Non]GNU-devel ELPA used this version string format:
>
>   VERSION.0-snapshotTIMESTAMP
>
> "snapshot" has to be used because "archive-contents" contains the
> version string in list format, e.g., (1 2 3 0 -4 20230405 111).  All of
> "snapshot", "cvs", "git", "bzr", "svn", "hg", "darcs" and "unknown" are
> encoded as -4 in the list format, but package-version-join maps it to
> "snapshot".
>
> That leads to very long version strings, which I assume is why Stefan
> switched to just
>
>   VERSION.0.TIMESTAMP
>
> The ".0" serves as a pseudo separator.  Currently "1.0-snapshot" is
> *smaller* than "1.0".  If we inject an additional ".0" then it is
> smaller than "N.0", but not smaller than "N".
>
> I would like to use a format that not only supported "pre-releases" but
> also "post-releases".  Debian uses "post-releases" to implement
> "debian-revision" [1]; we could use it to separate the "timestamp" part
> from "latest released upstream version" part, in snapshot release
> version strings.
>
> [1]: https://manpages.debian.org/wheezy/dpkg-dev/deb-version.5.en.html
>
> But again, maybe I am the only who feels a need to do something about
> that; in this context it is just an example of a change that would break
> backward compatibility; to demonstrate that some potential changes could
> not be made merely opt-in, but instead would have to be either rolled
> out to everyone or nobody.

I cannot comment on this, but "post-releases" (as opposed to
"pre-releases") sounds like just another entry in `version-regexp-alist'?

>      Jonas

So to summarise, I think an approximate plan would be useful:

1. Update package.el to be published on ELPA
2. (Optional) Update package-vc.el to be published on ELPA
3. Specify and implement a new package-archives format
4. Update package archives to generate the new archive format

Anything I am forgetting?