Hi,
Let me tell 2 real-life examples:
1. The one that made me find this issue:
My hobby project takes 45 seconds to compile, out of which the biggest .cc file that I modify most of the time is 15 seconds. In order to understand what's going on (it was not written by me, I'm just fixing bugs and adding new features) I often insert statements like printf("A\n"); so that I'll see the code is there. I copy-paste it to multiple places, replacing the letter with B, C etc.
Once I forgot to replace the letter. Saved, launched make, and immediately realized this mistake. So went back, fixed the file, saved, and re-launched make (probably just pressed the Up arrow and Enter, not even caring whether the previous make had already finished or not). I got hit by this race, and seeing only A's in the program's output mislead me into incorrectly thinking that B is not reached in the code, wasting a lot of time.
I prefer uninterrupted runs of development with full focus for maybe an hour or two at a time, re-compiling the app every minute or two in certain phases of development. Standing up and grabbing a coffee is not a viable option. (And yes, I do know about gdb, I just prefer printf-debugging whenever I can.)
2. Another example that also happens to me relatively frequently:
I download a mid-size piece of free software (let's say it compiles in 10 minutes) and apply a couple of patches to it. Let's assume applying the patches takes 5 minutes (I have to figure out the order, whether they're -p0 or -p1, some of them slightly conflict and I have to manually merge). If I perform these two steps sequentially, it's a total of 15 minutes. So, instead, what I do is: Start a make (or "make -k" preferably), and immediately afterwards start applying the patches. This way we use the CPU time and my time in parallel. Finally run "make" once again to catch up. This can reduce the total time in this example from 15 minutes to hardly more than 10 minutes. Alas, chances that I'll be bitten by this race condition is not negligible.
---
> I don't think make should be worried about this potential race; it's an
> obvious case of user error with an easy discipline for the user to apply
> to avoid any problems.
Is it really obvious? Is it documented somewhere? Do you really think pretty much all users of make think about it? I, for one, first used make (not just typed the command, but understood its basics, wrote a tiny Makefile for a tiny project) about 18-20 years ago, yet this race condition never occurred to me until recently.
Many years ago I was wondering what "make" would do if the source and destination files had the exact timestamp. I had better things to do than actually try it out or look it up, I just assumed that developers had put proper thought to it and found the solution that Just Works™. Nowadays with nanosecond timestamps I just don't care anymore.
Had I thought about it, I probably would have just guessed that I'm not the first one with this race condition problem, would have assumed that there's already a clever solution for this in place which, again, Just Works™. Seeing that there isn't any, I'm kindly asking you to think about it and come up with one.
> It's a rough-and-ready tool for developers, with
> no pretenses of being intended for use by those who don't think about
> how their tools work.
Do you really claim I should deeply think about all corner cases of the tools before using them? Given that "make" is just a tiny little fraction of all the tools I'm using, it's practically impossible. In fact, I think it's quite the opposite! The creators of such tools should do their best to make sure that the tools work reliably as expected.
I didn't think about the internals of make -- why should have I? I knew the goal it was serving: to create an up-to-date build in as little overall time as possible. With my particular development pattern it fails to achieve this goal. I don't think it's me who should've thought of this, nor that it's me who should change my workflow (touch the file after make completes, or not save while make is running, or start drinking coffee, or accept that I can not parallelize my time with the CPU's time, effectively taking away long minutes of my life many times for no good reason).
In my firm opinion, it should be make's job to do whatever it can to produce an up-to-date build under any circumstance. Hey, that's what make is all about!
> Of course, if a file-system had a separate "up to date at" file-stamp,
> that make can set (to the newest dependency's time-stamp) and later
> read, or if make were to store this same datum in a separate database,
> it would solve your problem. It would also make it possible to have
> make rules that tentatively build the new file, compare to existing,
> replace if (meaningfully) different but only revise "up to date at"
> otherwise.
Such a tiny "database" could indeed solve this problem without race condition. I would be more than happy to see this implemented! It wouldn't have to be a "database", just a simple ephemeral text file that's removed on clean exit of make, listing all the currently running rules that'd have to be rerun if this file exists on startup (meaning that the previous run was interrupted or crashed).
> As you note, every command that might be used in a make rule would need
> to do this "right", which is a prohibitive requirement.
>
> Requiring every command that could be used as a make rule to abide by
> this is, again, prohibitive. [...]
Some of these ideas I've thrown in might not work in practice, I agree.
Please note that I had a simple propsal: re-check the timestamps of input files when a rule completes and re-run (moving away or dating back the previous output file first) if any of them changed. This would not elimite the race condition completely, but would make the window magnitudes smaller (e.g. fraction of a millisecond instead of 15 seconds), and as such, would already be a great improvement!
cheers,
egmont