Re: `make check' overworks one core on my Core2 quad

[Graham Percival]

On Sun, Dec 13, 2009 at 2:26 PM, Han-Wen Nienhuys <address@hidden> wrote:
> On Sun, Dec 13, 2009 at 12:13 PM, Graham Percival
> <address@hidden> wrote:
>> On Sun, Dec 13, 2009 at 2:00 PM, Neil Puttock <address@hidden> wrote:
>>> 2009/12/13 Mark Polesky <address@hidden>:
>>>
>>>> Is there a way to improve this?  I don't want to put too
>>>> much extra stress on CPU1 if I run `make check' alot.  Or am
>>>> I being paranoid?
>>>
>>> make -j5 CPU_COUNT=5 check
>>
>> Be warned that sometimes lilypond-book has hash collisions in the
>> filename, which can lead to weird compile errors when one process
>> finished dealing with aa/lily-aaaa.ps (and thus deletes it), while
>> another process has finished generating aa/lily-aaaa.ps but hasn't
>> started running ps2pdf yet, and thus doesn't find the file that it
>> just wrote.
>
> Do you have real evidence for that?  We use 10 hex digits, yielding
> 2^40 combinations, so a 2^20 (one in a million) chance of collisions.

What happens if we include a regtest in the docs?  Or include a
snippet in the docs?  There's probably one or two .ly files in
Documentation/snippets/ that get compiled in 3 places (snippets.tely,
rhythms.itely, and expressive.itely, for example).  No matter what the
hash function is, hash(X) == hash(X).  Unless we use a pseudo-random
hash function.  :)

This bug occurred right after we changed the hash function to only
look at the snippet contents without including the preamble, which
makes it more likely that it's a "snippet frmo 1 manual coliding with
the same snippet in another manual, but with slightly different
compile options" thing.


> I think it is more likely that there is some kind of bug going on,
> given the state of the build system.

I've seen weird bugs that have to do with scheme not finding a file
(I'm not 90% certain it was a .ps file), when the .ly file was
definitely there.  Those bugs go away when I do a single-threaded bug.
 These multi-threaded bugs are not consistent.
(a few days ago, I tried replicating it in a tiny lilypond-book file
with 8 identical snippets; out of about 20 attempts to compile it, it
failed once)

I tried dumping a "print 'PANIC'" in lilypond-book in the function
that generates the name if it found an existing file there already.
It panicked.  I tried adding a "time.sleep(10)" if it detected the
file already existing, but that didn't end up helping.  (this was back
in Oct, not earlier this week.)


At this point, I gave up and changed gub/specs/lilypond-docs.py in GUB
to "parallel_build_broken = True", because there were much more
important problems in GUB at the time.

Cheers,
- Graham