Re: how to make dirs without races

[Tim Murphy]

Hi,

Using rules to make directories is a huge pain in the neck for many
reasons.  Directories behave more like files than one really wants
them to - any change to a directory by adding a file means it's time
is updated even if this is totally irrelevant, it will trigger a
rebuild of another file that has that directory as a prerequisite.
Order only prereqs don't help all that much - and of course I can't
remember why because that's GNUMake for you - it's so twisted and
complicated that I can't remember every last horrible windy road that
I have had to travel down.

I found it much simpler to create directories before the build.  There
are probably loads of ways to do this well and the one I came up with
was overcomplicated but it worked quickly on exceedingly large builds.
It used $(shell) but very sparingly so that it wasn't too slow.

https://bitbucket.org/tnmurphy/raptor/src/fbb2e624d320e5eabc0689105e2f2b80d131ca03/lib/flm/metaflm.mk?at=default#cl-116

The general plan was:
1. Paths were absolute
2. Use mkdir -p
3. Feed up to 30 directories into 1 $(shell mkdir   -p ..... ) call -
buffer until 30 are available.
4. Accept the fact that at every run of make you're going to attempt
to make directories - it's hardly noticeable.

for each target one might call:

$(call makepathfor,$(BITMAPHEADER))

Or for a logical collection of targets like all the targets that end
up in a library it's more efficient to append all required directories
to a variable like CREATABLEPATHS and then do it in one call:
$(call makepath,$(CREATABLEPATHS))


The buffering mechanism requires that you flush it at the end of your makefile:

$(call makepathfinalize)



Regards,

Tim

# Make the destination directory if necessary.  For some
# make engines we must do this outside the rule or they
# get confused by the apparent way in which different rules
# can create the same particular directory and they infer some kind
# of dependency.

# Buffering with repeat prevention, makes directories after every 30
calls. Any more might overload
# the createprocess limit on arguments.

GNUMKDIR:=/usr/bin/mkdir
makepathLIST:=

# add path to buffer. If the buffer has reached 30 paths then make
them and clear it.
define makepath_single
$(if $(findstring $1,$(makepathLIST)),,$(eval makepathLIST:=$(makepathLIST) $1))
$(if $(subst 30,,$(words $(makepathLIST))),,$(shell $(GNUMKDIR) -p
$(makepathLIST))$(eval makepathLIST:=))
endef

# Attempting to make things not already made is slow. It's something
to do with $(eval)
# or to do with creating huge numbers of TARGET_ variables?
# define makepath
# $(info makepath_start)$(foreach DIR,$1,$(if $(TARGET_$(1)),,$(call
makepath_single,$(DIR))$(eval TARGET_$(1):=1)))$(info makepath_end)
# endef

define makepath
$(strip $(foreach DIR,$(sort $1),$(call makepath_single,$(DIR))))
endef

define makepathfor
$(call makepath,$(dir $1))
endef

# Make any remaining paths in the path buffer
define makepathfinalise
$(strip $(if $(makepathLIST),$(shell $(GNUMKDIR) -p
$(makepathLIST))$(eval makepathLIST:=),))
endef




-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/