Re: non recursive makefiles

[Nicola Pero]

> Dealing with the last point first: I was not accusing the current
> maintainer of not maintaining it enough. It seemed a popular opinion
> that the current design isn't documented as much as it could be and I've
> struggled to link the Documentation with what exists at the minute,
> so I concluded that there are people who would like to help who don't
> understand the current code. There's more love which it could get.

Well, having read the rest of your email including the conclusion that you
think that the design is obviously all wrong and you know better makes
this initial captatio benevolentiae sound a bit flaky.

Anyway you have a point here that the documentation on the internals is
not much, and the code is often obscure (IMO mostly because the make
"language" is obscure).

On the plus side, it has never been documented so much, and the code has
never been easier :-)  It used to be so much worse in the past.

Everything has been reorganized so that there is actually a logic (even if
not always clearly documented) in the code, there are lot more useful
comments, and the code has been simplified a lot.



> What is the Master idea? Why can't we set the environment and then exec
> the user's preferred shell again instead of sourcing GNUstep.sh? Those
> are probably obvious questions and I didn't find answers.

Because every GNUmakefile starts with

include $(GNUSTEP_MAKFEILES)/common.make

and unless GNUSTEP_MAKEFILES is already set in the shell, you can't
bootstrap the gnustep-make code.  And if you have to set up stuff in your
shell anyway, why not doing the whole thing and sourcing GNUstep.sh and
getting all the variables ready - performance will benefit then. :-)



> > Well, I tried to explain in words.  You can't iterate in make.  You can't
> > execute pieces of code iteratively -- you can only execute them once, or
> > never.  this does not make it exactly easy to iterate over files, read
> > each of them, and then execute the same rules for each of them with
> > different variables.  You CAN NOT execute the same rules with different
> > variables in the same make invocation, which is what we want to do here.
> 
> Are you sure it's what we want to do? It's what we would want to do
> if we were using an imperative programming language like Objective-C
> to drive the build, but we agree that make is not a full programming
> language.
>
> According to its manual, GNU make "constructs a dependency graph of all
> the targets and their prerequisites." To that graph, the above situation
> needs to look like the final stage of building the files *depends
> on* having completed the previous stage for all files. Why can't that
> dependency be declared, rather than relying on implicit execution order
> in the version of make used?

I'm not sure what you mean, to "generate the dependency graph" you need to
have make parse and process the makefile fragment describing the rules
(which is what I mean by "executing the rules") multiple times, each for
every end target you want to build.  And every end target needs to get
customized rules for that target.  More customized than you seem to think,
you clearly haven't realized the complexity of building something like a
framework.


> I think this may be why using -j is breaking for some people. Have I 
> understood Helge's point correctly?

I'm not getting into any discussion about -j as I don't know remember
enough about it and I don't believe in discussing things you don't know
about.  I remember looking at it in the past when I had time to do it, and
concluding there was a fundamental problem with supporting it, and if you
ask me I will only quote my own informed opinion at the time.

I'm not sure it has to do with recursive invocations.

I'd like to spend some time researching it - cached answers need to be
revalidated after a while.  I still think that proper header preprocessing
would give a much bigger boost at a much smaller price.

 
> > Moreover, there are no local variables or rules.  Everything is global.  
> 
> Yes, this point is understood. There will be times when we need to
> do something specific with variables or have two subprojects that
> interfere with each other. Is this the general case, though? If not,
> can we accommodate the special cases without penalising the general?
> 
> In the few GNUmakefiles I've just looked at, it seems that things
> could be rewritten with prefixes or +=, or are already using prefixes.
> Some things, like APP_NAME, can already take a list. Most packages don't
> do anything more than the minimum of setting a few things and doing some
> includes - is that generally true?

Well - that's true for the GNUmakefile - that is the API.

What happens internally - the complexity of the rules and code used
internally to support your simple API - is entirely a different matter.

Ever tried to build a framework ?


 
> > Btw, how much better ... are you sure that that is what gives you slow
> > build times ? [...]
> 
> Even ignoring the point about -j, having to revisit every directory on
> a large project that was already built is a performance problem too. One
> feature of make is that it can do just the work needed to rebuild.

I don't buy this at all, the sandboxing of GNUmakefiles does well pay back 
for the performace you lose with the recursive invocation.

The fact that you're sandboxing allows you to use extremely complicated
make tricks which would be too difficult if they were included or used
multiple times across projects, and those make tricks let you avoid
spanning any subshell to compute anything.  You can create much more
complicated dependency graphs for each sandbox invocation, and support
many more complex features and complex intelligent behaviours and still be
fast than you would if you were using a single invocation.

And the performance difference you get nowadays on a new machine between a
recursive and iterative make invocation is small enough, and is getting
smaller and smaller.  The inclination to favour good design (sandboxing)  
over performance (a single big messy processing of everything in a single
go) gets bigger and bigger.

If you buy a new machine, the 'make' executable will be cached in memory
including all libraries and everything, the gnustep-make makefiles will be
in the kernel buffer, the GNUmakefiles too, the process can crunch through
them in a blazing sweep ... Even all the efforts I put into cutting away
unneeded shell invocations and pruning unused code and having gnustep-make
do the bare minimum it needs to do seem less useful now.  A 10%
improvement over 0.5 seconds is nothing.

But I acknolewdge that -j is an interesting issue, and it would really be
useful to support it.

 
> > > Yes, the subprojects that you use in one project have to be compatible,
> > > but that's already true.
> > No, it's not.  You can do
> > ADDITIONAL_OBJCFLAGS += -lPincoPallino
> > in one GNUmakefile, and that will *not* be seen by another GNUmakefile in 
> > another directory.
> 
> If you are messing with the general *CFLAGS for something which should
> not be passed to all final outputs, isn't that asking for trouble?
> Why not AppName_OBJCFLAGS += -lPincoPallino (or similar) if it's only
> for that app?

Well if you have 10 apps in a single GNUmakefile and you want to change
the OBJCFLAGS of all of them, you tend to use ADDITIONAL_OBJCFLAGS.

 
> > > Recursion doesn't change that fundamentally either.
> > Of course it does.  ADDITIONAL_OBJCFLAGS defined in subproject A have no
> > effect on ADDITIONAL_OBJCFLAGS defined in subproject B because they are
> > read in different, sandboxed, make invocations.
> 
> Depending on their relative positions in the hierarchy, subproject A can
> pollute subproject B even across make invocations. It's not global, but
> allowing it's still a bug, isn't it?
> 
> Some variables are reset with some make invocations, but it still
> alarms me to see it called "isolation" or "sandboxing".

I don't see your point.

First, all variables (except a few well-defined ones) are reset with
submake invocations.

Second, it's a hierarchical structure.  A makefile can affect its
children, but a child can not affect its parent or its sibling.

Third, even a parent affecting a children is discouraged because it makes
more difficult to run 'make' in the subdir and have it work indipendently.  
Most people don't do it, but you can do it if you want.



> > > Not isolation at all. Assuming isolation seems dangerous to me.
> > I don't agree.  "Isolation" is a basic design pattern of good software
> > architecture, and it looks like an essential requirement for a maintable,
> > usable system to assume isolation of subproject makefile code.
> 
> You are assuming isolation where it doesn't exist, which is the problem.
> How can a subproject's make be "isolated" from all other makefiles
> yet still allow settings passed from the environment?

Well how can an instance of NSString be isolated from another one and 
still be in the same program ?

The original environment is automatically passed around, local make
variables and rules are not.


 
> > > Recursion does allow "dirty" makefile practices to continue longer. Maybe
> > > sometimes recursion would be necessary, but I don't see why it seems to
> > > be the default for so many GNUstep packages. Is it just lack of time to
> > > hack an elegant solution, which is what your message suggests?
> > 
> > No - it's the way that make works that forces you to do this.
> 
> Are you sure? The gnustep-make DESIGN file looks ideal for using a
> straight make rather than recursing.

Somewhat the original author of the DESIGN document (which describes
mostly the requirements rather than the design) is also the one who shaped
the recursive-invocation design of the internals, and he did a fantastic
job.


> > make is supposed to let you define rules and targets and use them.  (btw,
> > generally, make does encourage you to use recursive invocations to iterate
> > over directories).
> 
> Make encourages you and then penalises you for doing it? I'd take
> the advice of such an awkward friend with caution ;-)

It encourages you because there are problems that have no other solution
than to use recursive invocations.

make doesn't support iteration, so you end up with recursion.

 
> Why do you think it's encouraged? It's documented, but that might be to
> stop people doing it badly. The case given as an example in the manual
> is fairly narrow. Other people are very scathing about it, such as
> "Recursive make Considered Harmful" by Peter Miller (1998), Journal of
> AUUG, http://aegis.sourceforge.net/auug97.pdf

Thanks - I'll read your article - that's really interesting.

 
> I think that gnustep-make is better than automake, but I wonder if more
> speed can be obtained by looking again for simplicity and seeing if
> any of the special case complexity can be refactored.  More efficient
> rebuilds and more use of -j seems desirable. The DESIGN looks sound,
> but I can't relate it to the code very well.
> 
> I acknowledge the edge case of incompatible projects, but I still
> haven't seen anything which looks like a general problem.  The problem
> causing iteration over files looked solvable by other means, but maybe
> I misunderstood it.

OK - whatever - I hope I have been detailed enough in my answers.

I'll leave this thread here as I am struggling with time and these emails
eat up a lot of time.