GOP-PROP 9: behavior of make doc

[Graham Percival]

The previous "build system" proposal, GOP-PROP 5, has been
cancelled.  That proposal failed to achieve clarity and consensus
within a month, mostly due to the proposal being too broad.
My apologies for failing to anticipate the level of disagreement;
I should have organized this discussion in a different way, or at
the very least cancelled the discussion earlier.

This new proposal focuses on "make doc" specifically.

http://lilypond.org/~graham/gop/gop_9.html


** Proposal summary

If there are build problems, then it should be easier to find out
why it’s failing. This will be achieved with log files, as well as
possibly including scripts which automatically display portions of
those log files for a failing build.

We will also add targets for building a specific manual (for
quick+easy checking of doc work), as well as for building all
documentation in a specific language (either English or a
translated language).


** Rationale

When the lilypond doc build breaks, it’s too hard to figure out
why it broke.

We see emails to lilypond-devel approximately once every four
months about broken doc builds. On a subjective note, Graham has
been the documentation editor since 2003, but even he cannot
reliably pinpoint the cause of a failing doc build within 10
minutes. We waste a ridiculous amount of time, effort, and
patience on doc build problems.


** Sea of output

Before any of the current work on reducing output from make, the
result of a “make doc” was over 500,000 lines of text. The prime
reason for the output being so verbose is that all the processes
that run as a result of the call to make echo their output to the
screen, often in verbose mode. Lilypond itself produces around
370,000 lines of output as a result of lilypond-book building all
the snippets.

Much of this output can be redirected to logfiles and so the
impossible-to-read clutter on the screen is cut out and could be
referred to later.


** Proposal details

When you run make doc,

    * All output will be saved to various log files, with the
      exception of output directly from make(1).

      Note that make(1) refers to a specific executable file on
unix computers, and is not a general term for the build system.

    * By default, no other output will be displayed on the
      console, with one exception: if a build fails, we might
      display some portion(s) of log file(s) which give useful
      clues about the reason for the failure.

      The user may optionally request additional output to be
printed; this is controlled with the VERBOSE=x flag. In such
cases, all output will still be written to log files; the console
output is strictly additional to the log files.

    * Logfiles from calling lilypond (as part of lilypond-book)
      will go in the relevant
      ‘build/out/lybook-db/12/lily-123456.log’ file. All other
      logfiles will go in the ‘build/logfiles/’ directory.
    * Both stderr and stdout will be saved in *.log. The order of
      lines from these streams should be preserved.
    * There will be no additional “progress messages” during the
      build process. If you run make --silent, a non-failing build
      should print absolutely nothing to the screen.
    * Assuming that the loglevels patch is accepted, lilypond
      (inside lilypond-book) will be run with –loglevel=WARN.
      http://codereview.appspot.com/4822055/
    * Ideally, a failing build should provide hints about the
      reason why it failed, or at least hints about which log
      file(s) to examine. 

If this proposal is accepted, none of these policies will be
assumed to apply to any other aspect of the build system. Policies
for any other aspect of the build system will be discussed in
separate proposals.


** Don’t cause more build problems

However, there is a danger in this approach, that vital error
messages can also be lost, thus preventing the cause of the
failure of a make being found. We therefore need to be
exceptionally careful to move cautiously, include plenty of tests,
and give time for people to experiment/find problems in each stage
before proceeding to the next stage.

This will be done by starting from individual lilypond calls
within lilypond-book, and slowly moving to “larger” targets of the
build system – after the individual lilypond calls are are
producing the appropriate amount of output and this is saved in
the right place and we can automatically isolate parts of a
failing build, we will work on lilypond-book in general, and only
then will we look at the build system itself.


** Implementation notes

There is an existing make variable QUIET_BUILD, which alter the
amount of output being displayed
(http://lilypond.org/doc/v2.15/Documentation/contributor/useful-make-variables
). We are not planning on keeping this make variable.

The standard way for GNU packages to give more output is with a
V=x option. Presumably this is done by increasing x? If we support
this option, we should still write log files; we would simply
print more of the info in those log files to screen.

The command tee may be useful to write to a file and display to
stdout (in the case of VERBOSE). 


Cheers,
- Graham