Re: updating many haskell packages

[Federico Beffa]

On Fri, Feb 17, 2017 at 6:47 PM, Troy Sankey <address@hidden> wrote:
> Quoting Federico Beffa (2017-02-17 03:00:04)
>> On Wed, Feb 15, 2017 at 8:31 PM, Troy Sankey <address@hidden> wrote:
>> > Quoting Troy Sankey (2017-02-06 16:00:29)
>> >> Quoting Federico Beffa (2017-02-06 15:34:47)
>> >> > I would consider a discrepancy between a cabal file on Hackage and the
>> >> > actual cabal file included in a tar archive a bug.  It may be helpful
>> >> > to report it to the author.
>> >>
>> >> I found this issue on the hackage github:
>> >>
>> >> https://github.com/haskell/hackage-server/issues/503
>> >>
>> >> It looks like it's a feature of hackage and/or cabal to allow multiple
>> >> revisions of cabal files per tarball release.  IIUC, the `cabal get`
>> >> utility takes care of automatically updating the cabal file to the
>> >> latest revision.
>> >>
>> >> I'll have to look more into this later.
>> >
>> > Indeed the .cabal files can be revised between releases.  Somehow the
>> > new revisions get pushed to the all-cabal-files repository [0] with an
>> > additional "x-revision" key [1], and Hackage picks up the new cabal
>> > file.  The `cabal` and `stack` utilities substitute the .cabal file from
>> > the release tarball with the revised .cabal file from Hackage if it is
>> > newer.
>> >
>> > In theory, anything in the .cabal file could be modified, not just
>> > dependency bounds.  I think guix should use the updated .cabal files for
>> > better consistency with the rest of the haskell ecosystem, and also for
>> > less per-package maintenance cost.
>> >
>> > How can guix adapt?  Should we somehow incorporate the all-cabal-files
>> > repository in the haskell build system, adding a post-unpack phase to
>> > substitute the entire .cabal file?
>>
>> Packages are build in isolated environments without network.
>> Therefore it's not possible to access Hackage with a phase of the
>> build system.  Even if it were possible it would be undesirable
>> because it would make things non reproducible.
>>
>> Files not included in the tar can be added as origin inputs to a
>> package definition (see the testsuite for GHC in the ghc package).
>> However, they are verified with hashes. Any change to those files will
>> break the package.
>>
>> Fede
>
> Forgive me if my understanding of build systems in Guix is flawed, but
> let me explain my idea with more detail:
>
> First, make a data-only package called "ghc-all-cabal-files" containing
> the checkout of a specific commit of the all-cabal-files repository from
> github.  We can periodically update this package, but there is no
> traditional "release"---we just keep pulling the HEAD of the hackage
> branch.  This package would then act as a helper package for the haskell
> build system---every haskell package should implicitly use this package
> as input.  Then we can write a post-unpack phase for the
> haskell-build-system which updates the unpacked .cabal file iff it finds
> a newer .cabal file in ghc-all-cabal-files (we know how to determine if
> the cabal file is newer: it will have a higher "x-revision" value, or
> that key will merely exist).
>
> One problem I have not fully solved is the technical debt associated
> with keeping the proposed ghc-all-cabal-files package up-to-date.  I
> believe updating it would require all haskell packages to be rebuilt.
> We could create a build system argument called use-newest-cabal-file to
> toggle the feature, in which case we would only switch it to #t if we
> already know the .cabal file to be stale.  Then only a small subset of
> packages would need to be rebuilt, and there is less technical debt than
> the current solution which involves monkey-patching every cabal file
> that needs it.

My worry with this approach is that every time that a single cabal
file in 'ghc-all-cabal-files' is updated all packages will fail to
build (the implicit input will fail). Given that there are several
thousands of cabal files, I suspect that this could occur quite often.

Maybe another approach could be: when we update the GHC packages we
follow a specific Stackage release. If a package tar includes an old
cabal file, we enable a 'jailbreak' flag such that the build system
clears version bounds. This of course relies on the fact that Stackage
verified that this is fine.

Regards,
Fede