Re: [Gnu-arch-users] Debian and package-framework

[Thomas Lord]

So, as an accidental side effect of this discussion, I wound up writing 
a little
essay about package framework.   I don't think it directly answers a
debian-maintainers concerns but I think the perspective might be
helpful anyway. 

Regards,
-t

                    Why We Use `package-framework'

 Sometimes people are surprised that we use `package-framework'
 rather than `autoconf' to configure and build our programs.

  Package-framework and the autoconf family of tools both make
 it easier to write Makefiles and configuration scripts, but
 they solve different problems.

  The autoconf family is best for bootstrapping the first few
 hard-to-port GNU programs in a new environment such as GNU
 Make, GNU Bash, and GCC.

  `package-framework' is better for most other programs:
 programs that are not so hard to port and that can count on
 the system to provide a Posix shell and GNU Make.

  * autoconf Solves a Chicken and Egg Problem;
     `package-framework' Makes a Chicken Omellete

  Consider the problem of writing a configure script and
 makefile for GNU Make and GNU Bash.  There's a
 chicken-and-egg problem.  The makefiles can't assume that GNU
 Make is already installed -- but the native versions of make
 found on many systems differ from one another greatly.
 Similarly, the configure scripts and the rules in makefiles
 can't assume that GNU Bash is already installed, yet there
 are awkward differences among the /bin/sh implementations
 found on many systems.  For those reasons, the autoconf tools
 implement "compilers" for makefiles and shell scripts.  These
 read high level descriptions and emit truly portable shell
 scripts and makefiles.

  Those compilers are tricky to write and maintain.  Worse,
 the autoconf tools encourage users to extend these compilers
 with new features, just for their own programs, and often
 users do this incorrectly -- producing a configure script or
 Makefile that will only work on a typical GNU/Linux system
 but might not work on, say, a FreeBSD system.  This is one of
 the ways that when people use autoconf other than to bootsrap
 those first few programs, the result can wind up being *less*
 portable: exactly the opposite of the reason to use autoconf
 in the first place!

  Package-framework, on the other hand, requires a standard
 Posix shell (Bash will do) and GNU Make.  Those are pretty
 small requirements.  In exchange, package-framework works
 without the need for shell script and makefile "compilers".
 This makes package-framework smaller, simpler, and easier to
 maintain.


  * `autoconf' Encourages Feature Tests,
     `package-framework' Encourages Standard Code

 Programs like GCC, Bash, and GNU Make were among the first
 written and distributed by the GNU project.  These programs
 could not assume that the GNU C library would be available
 because, at that time, it wouldn't even be complete for
 several more years.  Even today, no portable program is save
 to assume that the GNU C library is available.

 Back then many systems had C libraries and the usual header
 files but there was a problem.  Although POSIX standards
 existed, few systems implemented those standards perfectly.
 Some functions had different names on different systems.
 Some header files were in different places.  Some header
 files and functions were missing entirely.  Sometimes
 everything was in the right place, but a function was buggy,
 or had the wrong type.  All of these problems still happen
 today but to a much reduced extent: the POSIX standards are
 more widely and more accurately implemented.

 The variability between systems made it hard to write the
 first core GNU programs in a way that would compile in many
 different environments.  A simple fix for one environment
 could cause the build or the program to fail in another
 environment.  Generally speaking, people solve this kind of
 problem four different ways:

    1. conditional code based on system name
    2. conditional code based on parameters from the user
    3. conditional code based on feature tests -- probe the
       system during configuration to see what's there
    4. conservative coding and compatibility libraries

 The GNU tools use a mix of all of those but autoconf places a
 heavy emphasis on #3: feature tests.  (GCC, of course, must
 also use #1, especially learn about the CPU type.)

 Feature tests are generally better than making guesses based
 on the system name or requiring parameters from the user.
 The resulting pre-processor conditionals tend to be simpler.
 The result tends to be more robust and more easily ported.

 #4 is almost always the best choice *when the option is
 available*.  For the early GNU programs, the option was often
 not available.  There was no one portable way to do proper
 signal handling and subprocess management, for example, and
 no way to really solve that by writing a compatibility
 library.  So autoconf placed a heavy emphasis on feature
 tests and, to this day, encourages users to solve portability
 problems that way.

 `package-framework' is lucky to exist in a *slightly* simpler
 world.  It is much easier to write portable code, even for
 many low-level system operations, without any pre-processor
 conditionals at all.  So, while `package-framework' does have
 minimal support for feature tests, the emphasis is on
 conservative coding and compatibility libraries.

 * `autoconf' is Too Complicated
   `package-framework' is Too Simple

 The `autoconf' tools have been stretched far beyond their
 original mission to help porting the first few GNU tools.
 These days, they (it seems to us) are used for the vast
 majority of C and C++ programs in the free software world.
 `autoconf' has grown to be quite a bit larger than it
 started.  For some uses, it now depends on `perl'.  It has
 not gotten easier to maintain -- if anything, harder.  Many
 of us have experienced the frustration of `autoconf'-using
 programs that won't configure or compile correctly other than
 on a GNU/Linux system.  Many of us have experienced the
 frustration of having to have around multiple, incompatible
 versions of `autoconf' because some programs require one
 version while others require a different version.

  `package-framework', more or less, does one simple thing
 well.  It is already very simple and, even so: there is more
 code than needs to be there for the current functionality.
 If it changes in the future, is should become even simpler.

 But `package-framework' is in some ways a little bit *too*
 simple.  Right now, it offers no help at all building or
 linking against dynamically loaded libraries -- a major
 deficiency.  Additionally, it has no support for finding a
 required version of an installed library in the case where
 there may be more than one version installed.  These
 weaknesses are why, for example, a particular version of
 `libneon' is bundled with GNU Arch and why that library is
 statically linked to Arch.

 * A Possible Future

 `package-framework' generally serves us well and we think
 that its basic design is sound.  We believe that `autoconf'
 is over-stretched as it is popularly used and that it
 encourages people to use feature probes where more
 conservative coding practices would be a better solution.  We
 emphatically recognize that `autoconf' still does, and may
 "forever" play an important role in bootstrapping the most
 basic GNU system tools.

 The `package-framework' implementation today is a little bit
 messy.  It would be an improvement to carefully review the
 scripts and remove "dead code" and some minor "features" that
 serve no useful purpose.  It would be an improvement to
 carefully review the Makefile library and make similar
 changes.  The result would be a smaller, sleaker
 implementation of the current functionality.

 Next: `package-framework' needs support for dependencies on
 versioned libraries and for dynamic library loading.
 Versioned library support is a small matter of writing some
 new shell code -- there is no great mystery to it.  Dynamic
 loading is tricky because, indeed, this is an area where
 portability is hard to achieve.  It is not clear that
 adopting `libtool' directly is the right answer: `libtool'
 needs to be studied from a `package-framework' perspective
 and either adopted or reverse engineered and replaced.

 Next: There is a need for improved standards for packaging
 source distributions of software packages.  On all GNU
 systems, there should be a standard way to determine where
 to install programs and to tell what other programs have
 already been installed.   Configure/build tools are the
 right place to implement such standards.

 Finally, it is difficult to imagine an easy way to get many
 projects to switch to `package-framework' or to adopt new
 standards for source packages.  People have an awful lot
 invested in the tools they already use.  Vendors display only
 weak interest in that kind of standard.  We have many legacy
 packages, already very hard to configure and build, for which
 the effort of switching over would be monumental.
 Nevertheless, these are problems of the sort that can be
 meaningfully and usefully addressed in "one small step at a
 time" mode.   What will give people incentive to make the
 switch?  To have very good tools that make the switch easier
 to stomach, and to be offering very good benefits that result
 from switching.   Step 1: work on the tools, keeping the need
 for benefits in mind.




Sylvain Beucler wrote:
> On Mon, May 08, 2006 at 08:01:40PM +0200, Sylvain Beucler wrote:
>   
> > I have a few questions with regard to packaging / package-framework,
> > could you help us?
> >     
>
> I think I can answer some of them now, feel free to correct me :)
>
> - documentation: read comments in
>   src/build-tools/scripts/configure-dirs
>
> - cross-compilation: this would require passing --host/--build
>   ../configure options down to PLUGIN/AUTOCONF-based components, and
>   introducing a caching or best-guess mecanism in hackerlab's
>   compile-time tests afaics - so I can forget about it.
>
>
> - using externals libs instead of PLUGIN ones (for libneon): that one
> I still wonder about :) Is it supported by package-framework somehow,
> or will I have to trick the system? (eg. via some hand-made Makefile
> rules that would set gcc -I and ld -L appropriately?)
>
>
> Thanks!
>
>