Re: Proposal: using pkg-config for foreign dependencies in .egg-files

[Mario Domenech Goulart]

Hi Kristian,

On Sun, 15 Sep 2024 14:40:10 +0200 Kristian Lein-Mathisen 
<kristianlein@gmail.com> wrote:

> We've been discussing adding pkg-config support here at the Gosling
> 2024 chicken meetup. Here is a proposal.
>
> Using pkg-config can simplify adding foreign dependencies to
> eggs. Ideally, a single pkg-config identifier is sufficient to supply
> the chicken compiler with all the flags necessary to compile an egg
> which uses a foreign-dependency. The aim is to eliminate the need for
> custom-build scripts like
> [this](https://bugs.call-cc.org/browser/project/release/5/svn-client/trunk/build-svn-client).
>
> However, here are some of the downsides:
>
> (1) `pkg-config` may not be present on the build machine. We probably
> do not wish to have this as a hard dependency. However, it may be
> reasonable to depend on it to automatically build certain eggs with
> foreign dependencies. We try to adressed this below.
>
> (2) One concern is that the `pkg-config` identifiers may differ on
> different systems. If this turns out to be a problem in practice, I
> don't know if there is a clean solution for this.
>
> As far as I know, thre are two `pkg-config` implementations:
> pkg-config and pkgconf, the latter being a rewrite. The good news is
> that both provide a `pkg-config` executable which appear to be
> compatible.
>
> Taking the zstd egg as an example, here's a proposal of the new
> structure:
>
> ```
> ((synopsis "ZStandard bindings")
>  (author "Kristian Lein-Mathisen")
>  (category data)
>  (license "BSD")
>  (dependencies)
>  (test-dependencies test)
>  (components (extension zstd-loq-level
>                         (source "zstd-module.scm")
>                         (source-dependencies "zstd.scm")
>                         ;;(csc-options "-L" "-lzstd") ;; <-- must be removed 
> (would introduce duplicates)
>                         (foreign-dependencies (pkg-config libzstd)))
>              (extension zstd-nicer-api
>                         (source "zstd-nicer-api.scm"))
>              (program "zcompress")))
> ```
>
> So in practice,
>     (csc-options "-L" "-lzstd")
> turns into
>     (foreign-dependencies (pkg-config libzstd))
>
> Some remarks:
>
> - `foreign-dependencies` can now provide native library resolution using 
> `pkg-config`
> - `foreign-dependencies` can appear inside components and instead of top-level
> - `foreign-dependencies` is still allowed top-level for compatibility but is 
> still ignored (and should be
> deprecated?)
> - `foreign-dependencies` can contain a `(pkg-config pkg …)` specification
> - `foreign-dependencies` can be extended in the future to support other 
> specifications, e.g. `(nixos nixpackage …)`
> - `pkg` is passed to `pkg-config --libs --cflags $pkg` unless overridden
> - the output of `pkg-config --libs $pgk` can be overridden with environment 
> variables (even if `pkg-config` is
> present)
> - if `pkg-config` isn't present, we can provide an informative error say 
> which environment-variable to set
> - note: we assume `pkg` identifiers are representable as environment 
> variables (should be ok, they are already
> filenames)
>
> Here's an imaginary implementation:
>
> ```scheme
> (define (resolve-pkg-config pkg)
>   (let* ((var (conc "CHICKEN_INSTALL_PKG_CONFIG_" pkg))
>          (libs (get-environment-variable var)))
>   (if libs libs
>       (if (which "pkg-config")
>           (shell "pkg-config" "--libs" "--cflags" pkg)
>           (error (conc "pkg-config not found. install it, or set the 
> environment variable " var
>                        " to reflect the required CSC_OPTIONS.\n"
>                        "for example: export \"" var "=-L -l" (remove-suffix 
> pkg "lib") "\""))))))
> ```
>
> So, for example:
>
> ```
> # defaults work as expected
> $ pkg-config --libs libzstd
> -lzstd
> $ chicken-install zstd
> csc ... -lzstd ...
> $ ldd zstd.so
> libzstd.so.1 => ...
> ...
>
> $ apt remove pkg-config
> $ chicken-install zstd
> error: pkg-config not found. install it, or set the environment variable 
> CHICKEN_INSTALL_PKG_CONFIG_zstd.
> for example export "CHICKEN_INSTALL_PKG_CONFIG_zstd=-L -lzstd"
>
> # using pkg-config foreign-dependencies without pkg-config
> $ env CHICKEN_INSTALL_PKG_CONFIG_libzstd="-L -lzstd" chicken-install zstd
> ...
> $ ldd zstd.so
> libzstd.so.1 => ...
>
> # even with pkg-config installed, it can be useful to override foreign 
> dependencies:
> $ env "CHICKEN_INSTALL_PKG_CONFIG_zstd=-L -l/tmp/libzstd1.1.so" 
> chicken-install zstd
> ...
> $ ldd zstd.so
> libzstd1.1.so => /tmp/libzstd1.1.so
> ```

How about having something like `custom-config', which can be mapped
to a program that produces a list that can be read into the egg
specification tree?

Reusing your example:

```
((synopsis "ZStandard bindings")
 (author "Kristian Lein-Mathisen")
 (category data)
 (license "BSD")
 (dependencies)
 (test-dependencies test)
 (components
   (extension zstd-loq-level
              (source "zstd-module.scm")
              (source-dependencies "zstd.scm")
              (csc-options 
                (custom-config "custom-config.scm"))) ;; <- custom config 
program
             (extension zstd-nicer-api
             (source "zstd-nicer-api.scm"))
             (program "zcompress")))
```

`custom-config.scm' would be a program that writes a list of options
that would be used as `csc-options`.

With that, we are more flexible and future-proof regarding the needs
to adapt to different configuration mechanisms.  That also keeps the
core free of dependencies and assumptions about external configuration
tools.

To ease the process of coming up with custom-config programs, we can
rely on eggs, which can be used as `build-dependencies' in the egg
specification.  Such eggs can provide custom-config scripts that can
abstract all the dance needed to provide the right csc-options using
pkg-config, for example.

This would, however, introduce another loophole to the declarative
approach used by the egg format (the other one being custom-build),
but as it tends to happen with declarative approaches applied to
scenarios where flexibility is often needed, we usually have to choose
between limitations and complex specification formats to handle corner
cases.  Or break the declarative approach where convenient/necessary
(the proposed approach).

All the best.
Mario
-- 
http://parenteses.org/mario