FYI: FAQ chapter for the manual

[Alexandre Duret-Lutz]

I'm checking this in, on HEAD and branch-1-7.

2003-02-09  Alexandre Duret-Lutz  <address@hidden>

        * automake.texi (FAQ, CVS, maintainer-mode, wildcards)
        (distcleancheck): New nodes.

Index: automake.texi
===================================================================
RCS file: /cvs/automake/automake/automake.texi,v
retrieving revision 1.305.2.15
diff -u -r1.305.2.15 automake.texi
--- automake.texi       2 Feb 2003 10:45:41 -0000       1.305.2.15
+++ automake.texi       9 Feb 2003 20:53:48 -0000
@@ -134,6 +134,7 @@
 * Extending::                   Extending Automake
 * Distributing::                Distributing the Makefile.in
 * API versioning::              About compatibility between Automake versions
+* FAQ::                         Frequently Asked Questions
 * Macro and Variable Index::
 * General Index::
 @end menu
@@ -4537,7 +4538,9 @@
 your Makefiles cause some distributed files to be rebuilt when the user
 build the package.  (Think about the user missing the tool required to
 build the file; or if the required tool is built by your package,
-consider the cross-compilation case where it can't be run.)
+consider the cross-compilation case where it can't be run.)  There is
+a FAQ entry about this (@pxref{distcleancheck}), make sure you read it
+before playing with @code{distcleancheck_listfiles}.
 
 @code{distcheck} also checks that the @code{uninstall} target works
 properly, both for ordinary and @samp{DESTDIR} builds.  It does this
@@ -5334,7 +5337,7 @@
 package, regardless of the licensing you choose.
 
 
address@hidden API versioning, Macro and Variable Index, Distributing, Top
address@hidden API versioning, FAQ, Distributing, Top
 @chapter Automake API versioning
 
 New Automake releases usually include bug fixes and new features.
@@ -5399,8 +5402,487 @@
 @email{automake@@gnu.org} and try to get it documented and exercised by
 the test-suite.
 
address@hidden FAQ, Macro and Variable Index, API versioning, Top
address@hidden Frequently Asked Questions about Automake
+
+This chapters covers some questions that often come up on the mailing
+lists.
+
address@hidden
+* CVS::                         CVS and generated files
+* maintainer-mode::             missing and AM_MAINTAINER_MODE
+* wildcards::                   Why doesn't Automake support wildcards?
+* distcleancheck::              Files left in build directory after distclean
address@hidden menu
+
address@hidden CVS, maintainer-mode, FAQ, FAQ
address@hidden CVS and generated files
+
address@hidden Background: distributed generated files
address@hidden generated files, distributed
address@hidden rebuild rules
+
+Packages made with Autoconf and Automake ship with some generated
+files like @file{configure} or @file{Makefile.in}.  These files were
+generated on the developer's host and are distributed so that
+end-users do not have to install the maintainer tools required to
+rebuild them.  Other generated files like Lex scanners, Yacc parsers,
+or Info documentation, are usually distributed on similar grounds.
+
+Automake output rules in @file{Makefile}s to rebuild these files.  For
+instance @command{make} will run @command{autoconf} to rebuild
address@hidden whenever @file{configure.in} is changed.  This makes
+development safer by ensuring a @file{configure} is never out-of-date
+with respect to @file{configure.in}.
+
+As generated files shipped in packages are up-to-date, and because
address@hidden preserves timestamps, these rebuild rules are not
+triggered when a user unpacks and builds a package.
+
address@hidden Background: CVS and timestamps
address@hidden timestamps and CVS
address@hidden CVS and timestamps
+
+Unless you use CVS keywords (in which case files must be updated at
+commit time), CVS preserves timestamp during @code{cvs commit} and
address@hidden import -d} operations.
+
+When you check out a file using @code{cvs checkout} its timestamp is
+set to that of the revision which is being checked out.
+
+However, during @command{cvs update}, files will have the date of the
+update, not the original timestamp of this revision.  This is meant to
+make sure that @command{make} notices sources files have been updated.
+
+This timestamp shift is troublesome when both sources and generated
+files are kept under CVS.  Because CVS processes files in alphabetical
+order, @file{configure.in} will appear older than @file{configure}
+after a @command{cvs update} that updates both files, even if
address@hidden was newer than @file{configure.in} when it was
+checked in.  Calling @code{make} will then trigger a spurious rebuild
+of @file{configure}.
+
address@hidden Living with CVS in Autoconfiscated projects
address@hidden CVS and generated files
address@hidden generated files and CVS
+
+There are basically two clans amongst maintainers: those who keep all
+distributed files under CVS, including generated files, and those who
+keep generated files @emph{out} of CVS.
+
address@hidden All files in CVS
+
address@hidden @bullet
address@hidden
+The CVS repository contains all distributed files so you know exactly
+what is distributed, and you can checkout any prior version entirely.
+
address@hidden
+Maintainers can see how generated files evolve (for instance you can
+see what happens to your @file{Makefile.in}s when you upgrade Automake
+and make sure they look OK).
+
address@hidden
+Users do not need the autotools to build a checkout of the project, it
+works just like a released tarball.
+
address@hidden
+If users use @command{cvs update} to update their copy, instead of
address@hidden checkout} to fetch a fresh one, timestamps will be
+inaccurate.  Some rebuild rules will be triggered and attempt to
+run developer tools such as @command{autoconf} or @command{automake}.
+
+Actually, calls to such tools are all wrapped into a call to the
address@hidden script discussed later (@pxref{maintainer-mode}).
address@hidden will take care of fixing the timestamps when these
+tools are not installed, so that the build can continue.
+
address@hidden
+In distributed development, developers are likely to have different
+version of the maintainer tools installed.  In this case rebuilds
+triggered by timestamp lossage will lead to spurious changes
+to generated files.  There are several solutions to this:
+
address@hidden
address@hidden
+All developers should use the same versions, so that the rebuilt files
+are identical to files in CVS.  (This starts to be difficult when each
+project you work on uses different versions.)
address@hidden
+Or people use a script to fix the timestamp after a checkout (the GCC
+folks have such a script).
address@hidden
+Or @file{configure.in} uses @code{AM_MAINTAINER_MODE}, which will
+disable all these rebuild rules by default.  This is further discussed
+in @ref{maintainer-mode}.
address@hidden itemize
+
address@hidden
+Although we focused on spurious rebuilds, the converse can also
+happen.  CVS's timestamp handling can also let you think an
+out-of-date file is up-to-date.
+
+For instance, suppose a developer has modified @file{Makefile.am} and
+rebuilt @file{Makefile.in}, and then decide to do a last-minute change
+to @file{Makefile.am} right before checking in both files (without
+rebuilding @file{Makefile.in} to account for the change).
+
+This last change to @file{Makefile.am} make the copy of
address@hidden out-of-date.  Since CVS processes files
+alphabetically, when another developer @code{cvs update} his or her
+tree, @file{Makefile.in} will happen to be newer than
address@hidden  This other developer will not see
address@hidden is out-of-date.
+
address@hidden itemize
+
address@hidden Generated files out of CVS
+
+One way to get CVS and @code{make} working peacefully is to never
+store generated files in CVS, i.e., do not CVS-control files which are
address@hidden targets (or @emph{derived} files in Make terminology).
+
+This way developers are not annoyed by changes to generated files.  It
+does not matter if they all have different versions (assuming they are
+compatible, of course).  And finally, timestamps are not lost, changes
+to sources files can't be missed as in the
address@hidden/@file{Makefile.in} example discussed earlier.
+
+The drawback is that the CVS repository is not an exact copy of what
+is distributed and that users now need to install various development
+tools (maybe even specific versions) before they can build a checkout.
+But, after all, CVS's job is versioning, not distribution.
+
+Allowing developers to use different versions of their tools can also
+hide bugs during distributed development.  Indeed, developers will be
+using (hence testing) their own generated files, instead of the
+generated files that will be released actually.  The developer who
+prepares the tarball might be using a version of the tool that
+produces bogus output (for instance a non-portable C file), something
+other developers could have noticed if they weren't using there own
+versions of this tool.
+
address@hidden Third-party files
address@hidden CVS and third-party files
address@hidden third-party files and CVS
+
+Another class of files not discussed here (because they do not cause
+timestamp issues) are files which are shipped with a package, but
+maintained elsewhere.  For instance tools like @command{gettextize}
+and @command{autopoint} (from Gettext) or @command{libtoolize} (from
+Libtool), will install or update files in your package.
+
+These files, whether they are kept under CVS or not, raise similar
+concerns about version mismatch between developers' tools.  The
+Gettext manual has a section about this, see @ref{CVS Issues, CVS
+Issues, Integrating with CVS, gettext, GNU gettext tools}.
+
address@hidden maintainer-mode, wildcards, CVS, FAQ
address@hidden @command{missing} and @code{AM_MAINTAINER_MODE}
+
address@hidden @command{missing}
address@hidden missing, purpose
+
+The @command{missing} script is a wrapper around several maintainer
+tools, designed to warn users if a maintainer tool is required but
+missing.  Typical maintainer tools are @command{autoconf},
address@hidden, @command{bison}, etc.  Because file generated by
+these tools are shipped with the other sources of a package, these
+tools shouldn't be required during a user build and they are not
+checked for in @file{configure}.
+
+However, if for some reason a rebuild rule is triggered and involves a
+missing tool, @command{missing} will notice it and warn the user.
+Besides the warning, when a tool is missing, @command{missing} will
+attempt to fix timestamps in a way which allow the build to continue.
+For instance @command{missing} will touch @file{configure} if
address@hidden is not installed.  When all distributed files are
+kept under CVS, this feature of @command{missing} allows user
address@hidden no maintainer tools} to build a package off CVS, bypassing
+any timestamp inconsistency implied by @code{cvs update}.
+
+If the required tool is installed, @command{missing} will run it and
+won't attempt to continue after failures.  This is correct during
+development: developers love fixing failures.  However, users with
+wrong versions of maintainer tools may get an error when the rebuild
+rule is spuriously triggered, halting the build.  This failure to let
+the build continue is one of the arguments of the
address@hidden advocators.
+
address@hidden @code{AM_MAINTAINER_MODE}
address@hidden AM_MAINTAINER_MODE, purpose
address@hidden AM_MAINTAINER_MODE
+
address@hidden disables the so called "rebuild rules" by
+default.  If you have @code{AM_MAINTAINER_MODE} in
address@hidden, and run @code{./configure && make}, then
address@hidden will *never* attempt to rebuilt @file{configure},
address@hidden, Lex or Yacc outputs, etc.  I.e., this disables
+build rules for files which are usually distributed and that users
+should normally not have to update.
+
+If you run @code{./configure --enable-maintainer-mode}, then these
+rebuild rules will be active.
+
+People use @code{AM_MAINTAINER_MODE} either because they do want their
+users (or themselves) annoyed by timestamps lossage (@pxref{CVS}), or
+because they simply can't stand the rebuild rules and prefer running
+maintainer tools explicitly.
+
address@hidden also allows you to disable some custom build
+rules conditionally.  Some developers use this feature to disable
+rules that need exotic tools that users may not have available.
+
+Several years ago François Pinard pointed out several arguments
+against @code{AM_MAINTAINER_MODE}.  Most of them relate to insecurity.
+By removing dependencies you get non-dependable builds: change to
+sources files can have no-effect on generated files and this can be
+very confusing when unnoticed.  He adds that security shouldn't be
+reserved to maintainers (what @code{--enable-maintainer-mode}
+suggests), on the contrary.  If one user has to modify a
address@hidden, then either @file{Makefile.in} should be updated
+or a warning should be output (this is what Automake uses
address@hidden for) but the last thing you want is that nothing
+happens and the user doesn't notice it (this is what happens when
+rebuild rules are disabled by @code{AM_MAINTAINER_MODE}).
+
+Jim Meyering, the inventor of the @code{AM_MAINTAINER_MODE} macro was
+swayed by François's arguments, and got rid of
address@hidden in all of his packages.
+
+Still many people continue to use @code{AM_MAINTAINER_MODE}, because
+it helps them working on projects where all files are kept under CVS,
+and because @command{missing} isn't enough if you have the wrong
+version of the tools.
+
+
address@hidden wildcards, distcleancheck, maintainer-mode, FAQ
address@hidden Why doesn't Automake support wildcards?
address@hidden wildcards
+
+Developers are lazy.  They often would like to use wildcards in
address@hidden, so they don't need to remember they have to
+update @file{Makefile.am}s every time they add, delete, or rename a
+file.
+
+There are several objections to this:
address@hidden
address@hidden
+When using CVS (or similar) developers need to remember they have to
+run @code{cvs add} or @code{cvs rm} anyway.  Updating
address@hidden accordingly quickly becomes a reflex.
+
+Conversely, if your application doesn't compile
+because you forgot to add a file in @file{Makefile.am}, it will help
+you remember to @code{cvs add} it.
+
address@hidden
+Using wildcards makes easy to distribute files by mistake.  For
+instance some code a developer is experimenting with (a test case,
+say) but which should not be part of the distribution.
+
address@hidden
+Using wildcards it's easy to omit some files by mistake.  For
+instance one developer creates a new file, uses it at many places,
+but forget to commit it.  Another developer then checkout the
+incomplete project and is able to run `make dist' successfully,
+even though a file is missing.
+
address@hidden
+Listing files, you control *exactly* what you distribute.
+If some file that should be distributed is missing from your
+tree, @code{make dist} will complain.  Besides, you don't distribute
+more than what you listed.
+
address@hidden
+Finally it's really hard to @file{forget} adding a file to
address@hidden, because if you don't add it, it doesn't get
+compiled nor installed, so you can't even test it.
address@hidden itemize
+
+Still, these are philosophic objections, and as such you may disagree,
+or find enough value in wildcards to dismiss all of them.  Before you
+start writing a patch against Automake to teach it about wildcards,
+let's see the the main technical issue: portability.
+
+Although @code{$(wildcard ...)} works with GNU @command{make}, it is
+not portable to other @command{make} implementations.
+
+The only way Automake could support @command{$(wildcard ...)} is by
+expending @command{$(wildcard ...)} when @command{automake} is run.
+Resulting @file{Makefile.in}s would be portable since they would
+list all files and not use @code{$(wildcard ...)}.  However that
+means developers need to remember they must run @code{automake} each
+time they add, delete, or rename files.
+
+Compared to editing @file{Makefile.am}, this really little win.  Sure,
+it's easier and faster to type @code{automake; make} than to type
address@hidden Makefile.am; make}.  But nobody bothered enough to write a
+patch add support for this syntax.  Some people use scripts to
+generated file lists in @file{Makefile.am} or in separate
address@hidden fragments.
+
+Even if you don't care about portability, and are tempted to use
address@hidden(wildcard ...)} anyway because you target only GNU Make, you
+should know there are many places where Automake need to know exactly
+which files should be processed.  As Automake doesn't know how to
+expand @code{$(wildcard ...)}, you cannot use it in these places.
address@hidden(wildcard ...)} is a blackbox comparable to @code{AC_SUBST}ed
+variables as far Automake is concerned.
+
+You can get warnings about @code{$(wildcard ...}) constructs using the
address@hidden flag.
+
address@hidden distcleancheck,  , wildcards, FAQ
address@hidden Files left in build directory after distclean
address@hidden distclean, diagnostic
address@hidden dependencies and distributed files
address@hidden distclean
address@hidden distcleancheck
+
+This is a diagnostic you might encounter while running @code{make
+distcheck}.
+
+As explained in @ref{Dist}, @code{make distcheck} attempts to build
+and check your package for errors like this one.
+
address@hidden distcheck} will perform a @code{VPATH} build of your
+package, and then call @code{make distclean}.  Files left in the build
+directory after @code{make distclean} has run are listed after this
+error.
+
+This diagnostic really covers two kinds of errors:
+
address@hidden @bullet
address@hidden
+files that are forgotten by distclean;
address@hidden
+distributed files that are erroneously rebuilt.
address@hidden itemize
+
+The former left-over files are not distributed, so the fix is to mark
+them for cleaning (@pxref{Clean}), this is obvious and doesn't deserve
+more explanations.
+
+The latter bug is not always easy to understand and fix, so let's
+proceed with an example.  Suppose our package contains a program for
+which we want to build a man page using @command{help2man}.  GNU
address@hidden produces simple manual pages from the @code{--help}
+and @code{--version} output of other commands (@pxref{Top, , Overview,
+help2man, The Help2man Manual}).  Because we don't want our users to
+install @command{help2man}, we decide to distribute the generated man
+page using the following setup.
+
address@hidden
+# This Makefile.am is bogus.
+bin_PROGRAMS = foo
+foo_SOURCES = foo.c
+dist_man_MANS = foo.1
+
+foo.1: foo$(EXEEXT)
+       help2man --output=foo.1 ./foo$(EXEEXT)
address@hidden example
+
+This will effectively distribute the man page.  However,
address@hidden distcheck} will fail with:
+
address@hidden
+ERROR: files left in build directory after distclean:
+./foo.1
address@hidden example
+
+Why was @file{foo.1} rebuilt?  Because although distributed,
address@hidden depends on a non-distributed built file:
address@hidden(EXEEXT)}.  @file{foo$(EXEEXT)} is built by the user, so it
+will always appear to be newer than the distributed @file{foo.1}.
+
address@hidden distcheck} caught an inconsistency in our package.  Our
+intent was to distribute @file{foo.1} so users do not need installing
address@hidden, however since this our rule causes this file to be
+always rebuilt, users @emph{do} need @command{help2man}.  Either we
+should ensure that @file{foo.1} is not rebuilt by users, or there is
+no point in distributing @file{foo.1}.
+
+More generally, the rule is that distributed files should never depend
+on non-distributed built files.  If you distribute something
+generated, distribute its sources.
+
+One way to fix the above example, while still distributing
address@hidden is to not depend on @file{foo$(EXEEXT)}.  For instance,
+assuming @command{foo --version} and @command{foo --help} do not
+change unless @file{foo.c} or @file{configure.ac} change, we could
+write the following @file{Makefile.am}:
+
address@hidden
+bin_PROGRAMS = foo
+foo_SOURCES = foo.c
+dist_man_MANS = foo.1
+
+foo.1: foo.c $(top_srcdir)/configure.ac
+        $(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
+       help2man --output=foo.1 ./foo$(EXEEXT)
address@hidden example
+
+This way, @file{foo.1} will not get rebuilt every time
address@hidden(EXEEXT)} changes.  The @command{make} call makes sure
address@hidden(EXEEXT)} is up-to-date before @command{help2man}.  Another
+way to ensure this would be to use separate directories for binaries
+and man pages, and set @code{SUBDIRS} so that binaries are built
+before man pages.
+
+We could also decide not to distribute @file{foo.1}.  In
+this case it's fine to have @file{foo.1} dependent upon
address@hidden(EXEEXT)}, since both will have to be rebuilt.
+However it would be impossible to build the package in a
+cross-compilation, because building @file{foo.1} involves
+an @emph{execution} of @file{foo$(EXEEXT)}.
+
+Another context where such errors are common is when distributed files
+are built by tools which are built by the package.  The pattern is similar:
+
address@hidden
+distributed-file: built-tools distributed-sources
+        build-command
address@hidden example
+
address@hidden
+should be changed to
+
address@hidden
+distributed-file: distributed-sources
+        $(MAKE) $(AM_MAKEFLAGS) built-tools
+        build-command
address@hidden example
+
address@hidden
+or you could choose not to distribute @file{distributed-file}, if
+cross-compilation does not matter.
+
+The points made through these examples are worth a summary:
+
address@hidden
address@hidden
address@hidden
+Distributed files should never depend upon non-distributed built
+files.
address@hidden
+Distributed files should be distributed will all their dependencies.
address@hidden
+If a file is @emph{intended} be rebuilt by users, there is no point in
+distributing it.
address@hidden itemize
address@hidden cartouche
+
address@hidden distcleancheck_listfiles
+For desperate cases, it's always possible to disable this check by
+setting @code{distcleancheck_listfiles} as documented in @ref{Dist}.
+Make sure you do understand the reason why @code{make distcheck}
+complains before you do this.  @code{distcleancheck_listfiles} is a
+way to @emph{hide} errors, not to fix them.  There is always better to
+do.
+
 @page
address@hidden Macro and Variable Index, General Index, API versioning, Top
address@hidden Macro and Variable Index, General Index, FAQ, Top
 @unnumbered Macro and Variable Index
 
 @printindex vr
-- 
Alexandre Duret-Lutz