Re: Idea: Add .COMMANDCHANGE and .CACHE

[Britton Kerin]

> I think .COMMANDCHANGE is not complex.
> Since it caches the expanded command, at worst it will re-run a script
> when before it would not have done so.
> So it's relatively low risk; at worst, it'll run a command more often than 
> before.
> It won't rerun if only an environment variable changes, but that is *already* 
> true.

True but it's a sort of misleading way to look at things.  With .COMMANDCHANGE
you're effectively saying "add this thing and you can efficiently intermingle
build system changes with other development without worry".  Fail to deliver on
that promise and the feature will likely cause net pain.

> I'm hoping that there's a better name than .COMMANDCHANGE.
>
> I *agree* that .CACHE is more complex to use.
> Its implementation is NOT complex - indeed, it can be
> implemented in relatively few lines of code.
> The challenge is that while .COMMANDCHANGE causes execution
> to happen more than now, .CACHE can cause an execution to be
> *completely* skipped and its cache results would be used instead.
>
> The risk of .CACHE is that it can produce "wrong results", and
> that is a valid concern.  In any cache system (including this one)
> the cache key must be calculated from an absolutely complete
> set of all relevant inputs, and the output (target) list *must*
> be a complete statement of the resulting targets.
> If there are multiple targets, using "&:" (not ":") may be critically
> important, not just something that can be caught on the next

I'm unlikely to ever use &, because I predate it by decades and  stamps/proxies
have always worked fine for me (I've never been tempted to edit or delete a
proxied file and been burned as a result).  So it would be nice if
.COMMANDCHANGE/.CACHE still worked without &, and I think to be really correct
it must since nothing in existing make forbids these common arrangements.

> run of "make".  If environment variables' settings matter, they
> need to be included in the inputs.  For example, the environment variables
> could be expanded somewhere in the command (e..g, @printf '' "$MYVAR").
> That means that .CACHE makes the system even more dependent
> on accurate statements about targets & prerequisites.
> For example: I didn't include read/write/execute permissions of prerequisites
> in the inputs to the cache keys on purpose.  However, technically their
> permission values can have an effect, and the current .CACHE doesn't handle 
> that.
>
> I'd love to hear ways of reducing the risks of .CACHE.  Some ideas:
> 1. We could define a "CACHEENV" variable, a list of all environment variables
>    whose expansions would be included in the cache key.
>    Adding a few environment variables to this list would mean that changing 
> any
>    of them would cause current cached values to be skipped.

To be conservative and automatic I would probably include the entire
environment and then explicitly subtract out e.g. X/DBUS garbage if feature was
provided to do so.  I guess this could be done with text functions easily enough
too though.

 > 2. When multiple targets are listed in a traditional rule (":" not "&:"),
>    after running a rule in non-parallel mode, check to see if any of the
>    other targets were modified.  If they were, warn or error out & explain 
> the problem
>    ("use &: instead of :, or create an intermediary value on line XYZ").
>    This might be a good idea anyway, even if .CACHE isn't added...
>    I just got bit by this mistake yesterday!  Automatically detecting common
>    mistakes, and warning about them, is often a great idea.

Again, I think what you really want is a key derived from all the recipes in
the entire dependency DAG of the target.  I think this picks up the case you're
considering here, plus others where significant side effects exist that aren't
explicitly mentioned in the DAG at all.

Britton