Re: complex Makefile for a bootstrapped self-hosting translator [GCC MELT] ?

[Basile Starynkevitch]

On Mon, Jan 09, 2012 at 01:51:58PM -0500, Paul Smith wrote:
> On Mon, 2012-01-09 at 16:35 +0100, Basile Starynkevitch wrote:
> > I have big trouble with my make rules (for GNU make)
> > 
> > I detailed my issues in the below Stack Overflow question
> >    http://stackoverflow.com/q/8727896/841108
> > 
> > I need hints of how to debug parallel make, and how to improve
> > multi-stage building.
> 
> It would be simpler for us if your questions were more specifically
> targeted and detailed.  You have a lot of detail in describing your
> environment but not as much in describing the problems.


Thanks for replying. I'm giving more details here. I'm not repeating things 
already said 
in my StackOverflow question.

A prerequisite question is that I have no precise idea of when should timestamp 
files be used.
GCC Makefile-s have several of them, conventionally named s-* ; an example 
(e.g. near line 3873 
of file gcc/Makefile.in) is

   gtyp-input.list gtyp-real-input.list: s-gtyp-input ; @true
   s-gtyp-input: Makefile
        @: $(call write_entries_to_file,$(GTFILES),tmp-gi.list)
        $(SHELL) $(srcdir)/../move-if-change tmp-gi.list gtyp-input.list
        @: $(call write_entries_to_file,$(REALGTFILES),tmp-realgi.list)
        $(SHELL) $(srcdir)/../move-if-change tmp-realgi.list 
gtyp-real-input.list
        $(STAMP) s-gtyp-input

(STAMP is basically the "touch" command, or "echo timestamp >")

A long comment (near line 1572) is explaining why stampfiles are needed:

# On the use of stamps:
# Consider the example of tree-check.h. It is constructed with build/gencheck.
# A simple rule to build tree-check.h would be
# tree-check.h: build/gencheck$(build_exeext)
#       $(RUN_GEN) build/gencheck$(build_exeext) > tree-check.h
#
# but tree-check.h doesn't change every time gencheck changes. It would the
# nice if targets that depend on tree-check.h wouldn't be rebuild
# unnecessarily when tree-check.h is unchanged. To make this, tree-check.h
# must not be overwritten with a identical copy. One solution is to use a
# temporary file
# tree-check.h: build/gencheck$(build_exeext)
#       $(RUN_GEN) build/gencheck$(build_exeext) > tmp-check.h
#       $(SHELL) $(srcdir)/../move-if-change tmp-check.h tree-check.h
#
# This solution has a different problem. Since the time stamp of tree-check.h
# is unchanged, make will try to update tree-check.h every time it runs.
# To prevent this, one can add a stamp
# tree-check.h: s-check
# s-check : build/gencheck$(build_exeext)
#       $(RUN_GEN) build/gencheck$(build_exeext) > tmp-check.h
#       $(SHELL) $(srcdir)/../move-if-change tmp-check.h tree-check.h
#       $(STAMP) s-check
#
# The problem with this solution is that make thinks that tree-check.h is
# always unchanged. Make must be deceived into thinking that tree-check.h is
# rebuild by the "tree-check.h: s-check" rule. To do this, add a dummy command:
# tree-check.h: s-check; @true
# s-check : build/gencheck$(build_exeext)
#       $(RUN_GEN) build/gencheck$(build_exeext) > tmp-check.h
#       $(SHELL) $(srcdir)/../move-if-change tmp-check.h tree-check.h
#       $(STAMP) s-check
#
# This is what is done in this makefile. Note that mkconfig.sh has a
# move-if-change built-in


But I don't understand *when* should I use stampfiles (and my feeling is that 
the 
need of stampfiles is a symptom of some core feature missing in GNU make; I 
would 
believe I won't need them if I could use omake http://omake.metaprl.org/ or 
perhaps 
some better builder; but in the GCC world we are stuck to using GNU make)

 
> For debugging GNU make there are not many options other than the -d flag
> and the -p flag, both of which can be useful.  There is also remake
> ( http://bashdb.sourceforge.net/remake/ ) which might help, but may be
> more useful for incorrect makefiles rather than the higher-level issues
> you are raising.

Not really, my makefiles are buggy. They usually work.

 
> For debugging parallel build issues you normally just need to
> characterize what the error was ("file foo was built at the same time as
> file bar, but really foo needs bar to be created first") then add
> prerequisites to enforce those relationships.  Obviously that's a simple
> case and yours may be more complex, but you need to understand it in
> detail (then provide us with that detail).
> 
> For "do nothing" wait time, the output of -d is helpful and will
> probably show you that you have pattern rules that are being checked
> over and over.  The more pattern rules you have (especially "match
> anything" pattern rules) the longer make will take to compute and check
> all those options.  -d will show you all it's doing.


I forgot to explain, the do nothing 3 or 4 minutes time is not spent in make 
processes, 
but in actual commands which are not needed, and I have not a clear idea on how 
to detect 
that they are useless. The crux of my issues is that my commands should be 
conceptually
running when the contents of generated files are changed, not when their 
timestamp is 
changing.

> If you haven't done so yet (and you don't need the builtin rules which
> many makefiles don't), you can remove all builtin implicit rules by
> adding this to your makefile:
> 
>         .SUFFIXES:
>         %:: %,v
>         %:: RCS/%,v
>         %:: RCS/%
>         %:: s.%
>         %:: SCCS/s.%

Why should that help in practice?

> 
> Even if you do need some builtin rules it's often better to remove them
> all then add back the ones you want explicitly.

Why don't usual Makefile-s (in particular autotools generated ones don't have 
these)?

Thanks for reading.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***