[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [bug #46242] Race condition when input file is updated while compili

From: David Boyce
Subject: Re: [bug #46242] Race condition when input file is updated while compiling
Date: Tue, 20 Oct 2015 12:57:53 -0700

My view is that this is really a variant of the longstanding
discussion about getting make to not rely on timestamps at all and use
MD5 or similar digital signatures instead. There are a number of
reasons why timestamps can be insufficient beside this race condition,
so I think it might be better to put effort into getting that working
than doing more work on the timestamp implementation.

Note that there are published mechanisms for using checksums with
existing make. For instance

David Boyce

On Tue, Oct 20, 2015 at 11:36 AM, Egmont Koblinger <address@hidden> wrote:
> Hi,
> Let me tell 2 real-life examples:
> 1. The one that made me find this issue:
> My hobby project takes 45 seconds to compile, out of which the biggest .cc
> file that I modify most of the time is 15 seconds.  In order to understand
> what's going on (it was not written by me, I'm just fixing bugs and adding
> new features) I often insert statements like printf("A\n"); so that I'll see
> the code is there.  I copy-paste it to multiple places, replacing the letter
> with B, C etc.
> Once I forgot to replace the letter. Saved, launched make, and immediately
> realized this mistake.  So went back, fixed the file, saved, and re-launched
> make (probably just pressed the Up arrow and Enter, not even caring whether
> the previous make had already finished or not).  I got hit by this race, and
> seeing only A's in the program's output mislead me into incorrectly thinking
> that B is not reached in the code, wasting a lot of time.
> I prefer uninterrupted runs of development with full focus for maybe an hour
> or two at a time, re-compiling the app every minute or two in certain phases
> of development.  Standing up and grabbing a coffee is not a viable option.
> (And yes, I do know about gdb, I just prefer printf-debugging whenever I
> can.)
> 2. Another example that also happens to me relatively frequently:
> I download a mid-size piece of free software (let's say it compiles in 10
> minutes) and apply a couple of patches to it.  Let's assume applying the
> patches takes 5 minutes (I have to figure out the order, whether they're -p0
> or -p1, some of them slightly conflict and I have to manually merge).  If I
> perform these two steps sequentially, it's a total of 15 minutes.  So,
> instead, what I do is:  Start a make (or "make -k" preferably), and
> immediately afterwards start applying the patches.  This way we use the CPU
> time and my time in parallel.  Finally run "make" once again to catch up.
> This can reduce the total time in this example from 15 minutes to hardly
> more than 10 minutes.  Alas, chances that I'll be bitten by this race
> condition is not negligible.
> ---
>> I don't think make should be worried about this potential race; it's an
>> obvious case of user error with an easy discipline for the user to apply
>> to avoid any problems.
> Is it really obvious?  Is it documented somewhere?  Do you really think
> pretty much all users of make think about it?  I, for one, first used make
> (not just typed the command, but understood its basics, wrote a tiny
> Makefile for a tiny project) about 18-20 years ago, yet this race condition
> never occurred to me until recently.
> Many years ago I was wondering what "make" would do if the source and
> destination files had the exact timestamp.  I had better things to do than
> actually try it out or look it up, I just assumed that developers had put
> proper thought to it and found the solution that Just Works™.  Nowadays with
> nanosecond timestamps I just don't care anymore.
> Had I thought about it, I probably would have just guessed that I'm not the
> first one with this race condition problem, would have assumed that there's
> already a clever solution for this in place which, again, Just Works™.
> Seeing that there isn't any, I'm kindly asking you to think about it and
> come up with one.
>> It's a rough-and-ready tool for developers, with
>> no pretenses of being intended for use by those who don't think about
>> how their tools work.
> Do you really claim I should deeply think about all corner cases of the
> tools before using them?  Given that "make" is just a tiny little fraction
> of all the tools I'm using, it's practically impossible.  In fact, I think
> it's quite the opposite!  The creators of such tools should do their best to
> make sure that the tools work reliably as expected.
> I didn't think about the internals of make -- why should have I?  I knew the
> goal it was serving: to create an up-to-date build in as little overall time
> as possible.  With my particular development pattern it fails to achieve
> this goal.  I don't think it's me who should've thought of this, nor that
> it's me who should change my workflow (touch the file after make completes,
> or not save while make is running, or start drinking coffee, or accept that
> I can not parallelize my time with the CPU's time, effectively taking away
> long minutes of my life many times for no good reason).
> In my firm opinion, it should be make's job to do whatever it can to produce
> an up-to-date build under any circumstance.  Hey, that's what make is all
> about!
>> Of course, if a file-system had a separate "up to date at" file-stamp,
>> that make can set (to the newest dependency's time-stamp) and later
>> read, or if make were to store this same datum in a separate database,
>> it would solve your problem.  It would also make it possible to have
>> make rules that tentatively build the new file, compare to existing,
>> replace if (meaningfully) different but only revise "up to date at"
>> otherwise.
> Such a tiny "database" could indeed solve this problem without race
> condition.  I would be more than happy to see this implemented!  It wouldn't
> have to be a "database", just a simple ephemeral text file that's removed on
> clean exit of make, listing all the currently running rules that'd have to
> be rerun if this file exists on startup (meaning that the previous run was
> interrupted or crashed).
>> As you note, every command that might be used in a make rule would need
>> to do this "right", which is a prohibitive requirement.
>> Requiring every command that could be used as a make rule to abide by
>> this is, again, prohibitive.  [...]
> Some of these ideas I've thrown in might not work in practice, I agree.
> Please note that I had a simple propsal: re-check the timestamps of input
> files when a rule completes and re-run (moving away or dating back the
> previous output file first) if any of them changed.  This would not elimite
> the race condition completely, but would make the window magnitudes smaller
> (e.g. fraction of a millisecond instead of 15 seconds), and as such, would
> already be a great improvement!
> cheers,
> egmont
> _______________________________________________
> Bug-make mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/bug-make

reply via email to

[Prev in Thread] Current Thread [Next in Thread]