[bug #19448] Re-exec after "include file rebuild" is more dependent on filesystem timestamps than strictly necessary.

[Christopher Lester]

URL:
  <http://savannah.gnu.org/bugs/?19448>

                 Summary: Re-exec after "include file rebuild" is more
dependent on filesystem timestamps than strictly necessary.
                 Project: make
            Submitted by: kesterlester
            Submitted on: Thursday 03/29/2007 at 22:04
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 3.81
        Operating System: Any
           Fixed Release: None

    _______________________________________________________

Details:


I have attached a 22 line Makefile.
For the purposes of this document, always run it in the following way:

        "make clean ; make"

What you SHOULD see in the output is two occurrences of the word MOO
(corresponding to two re-generations of the included "foo.d" file) like
this:

-----------------------------------------------
address@hidden tmpfs]$ make clean ; make
Makefile:20: foo.d: No such file or directory
MOO
MOO
make: Nothing to be done for `all'.
address@hidden tmpfs]$ 
------------------------------------------------

which indeed is what you see when you run it in a directory with a good
timestamp resolution such as a "tmpfs" directory under linux made with eg
'mount tmpfs ~/tmpfs/ -t tmpfs'.  :)

Unfortunately, when you run the same "make clean ; make" on a typical linux
ext3 filesystem without extended attributes where stat returns timestamps
discretised at the 1-second level, you instead only see ONE occurrence of the
word MOO like this:

[------------------------------------------------
address@hidden v3]$ make clean ; make
Makefile:20: foo.d: No such file or directory
MOO
make: Nothing to be done for `all'.
address@hidden v3]$ 
------------------------------------------------

corresponging to only one re-generation of "foo.d". [Run the above a few
times - due to a race condition you may get lucky and see two MOOs a small
fraction of the time.]

It is possible to "fix" things (i.e. make "make" function the same
irrespective of the timestamp granularity of the underlying filesystem) by
slowing down the process of building "foo.d" by inserting an appropriately
long "sleep" after writing "foo.d". In the supplied makefile there is a
"usleep 1100000" commented out which, if un-commented, should provide enough
of a delay to fix things on a filesystem with a granularity of one second.

---------------------------------------------

Where is all this going?

(1) This is NOT supposed to be a complaint about make basing its rebuild
decisions on timestamps - that's in make's blood.  

(2) Make cannot be held responsible for things outside its control, such as
the poor time-resolution limitations in some filesystems.  Make has rules for
dealing with "equal" time stamps which it must follow carefully (and probably
does).

The point of the message is to highlight the freedom open in the
interpretation of the words

        "starts with a clean slate"

in the "remaking makefiles" part of the documentation:

   " ... After all makefiles have been checked, if any have actually been
changed, make starts with a clean slate and reads all the makefiles over
again. (It will also attempt to update each of them over again, but normally
this will not change them again, since they are already up to date.)"

I am lead to believe that the mechanism by which the current
make-implementation "starts with a clean slate" after rebuild of an include
file is literally by internally calling "exec" with itself and with its
original command-line-args.  [I've not checked the last statement in the
sourcecode, so it could be wrong, but the rest of this message doesn't rely
on it being true.]  Clearly there is a lot to be said for that approach from
the standpoint of simplicity!  That's one line of code and you're going to be
able to predict what it does.
If that is what make actually does (though the documentation doesn't REQUIRE
that "starts with a clean state" be implemented in exactly that way) then the
observed behaviour is probably 100% as expected.


Unfortunately this implementation choice means that the second (or n th)
invocation of make forgets the order in which any just-made files were made,
and is then at the mercy of the time-resolution of the filesystem of the
currenty directory to carry that information over from make-invocation-(n) to
make-invocation-(n+1), thereby introducing an un-necessary non-portability.

make-invocation-(n) may well know full-well that it generated file "a"
before/after file "b", and even that it intended to do so, even if the
filesystem timestamps are not capable of recording this and give them an
identical timestamp.

In my opinion, make-invocation-(n+1) could make use of information that was
known to make-invocation-(n) fully within the spirit and letter of "starts
with a clean slate" ... to my mind those words are supposed to signify that
make should re-process the whole makefile and any new includes and all
variables from scratch, and should use the best information it has about
file-update times etc, to build a new dependency graph and hit "go".  In my
opinion those same words are not supposed to FORCE make to throw away
information it already has which could be used as timestamp tie-breakers, and
put it at the mercy of a worse approximation sitting on the filesystem.

[ For example, a single invocation of a non-recursive make can build 1000
files lighting fast, and they might all get the same timestamp, but make
rightly doesn't care because it decided on their build order even before it
made the first of them.  Rightly so.  I'd like make-auto-re-exec to be "as
much like this as possible" within the constraints of the rules. ]

Naturally I admit that just because make "could" be implemented in some other
way than it is at present, while maintining better-than-present portability
across filesystems, does not mean that it would be easy to do so.

It would be way beyond me for a start!

But perhaps an expert could find some way of fixing things with a clever
trick such as passing make-invocation-(n+1) an extra argument like "-W
file.d" for every ".d" file that make thinks it just made which is forcing a
re-exec.   That idea is probably broken, but something related might work. 
I'm out of my depth.

Yours,

Christopher



    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Thursday 03/29/2007 at 22:04  Name: Makefile  Size: 712B   By:
kesterlester

<http://savannah.gnu.org/bugs/download.php?file_id=12337>

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?19448>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/