Re: Fail to merge staging - make doc is failing

[James]

Hello,

> On 15/09/13 15:05, David Kastrup wrote:
> > James <address@hidden> writes:
> >
> > > Hello,
> > >
> > > I cannot merge staging with master as there is an error during make 
> > > doc.
> > Currently the only commit in staging that is not in master is
> >
> > commit 6b79234fbfe9400cac355a7b3d7da90532d82522 (origin/staging, 
> > issue3554)
> > Author: David Kastrup <address@hidden>
> > Date:   Sun Sep 15 11:01:06 2013 +0200
> >
> >      Issue 3554: Avoid empty grammar rules using @$ for getting a 
> > source location
> >           Several rules use @$ for getting the source position from 
> > an empty
> >      rule, but Bison does not even provide a source position for empty
> >      rules.
> >           This patch fixes most cases by letting different non-empty 
> > rules
> >      assign the respective source positions.  The case of #{ #} 
> > cannot be
> >      fixed in that manner since there is not a single token providing a
> >      source position in the whole production.  Fixing this particular 
> > case
> >      would require a different technique.
> >
> > And that commit does not seem very likely to cause that symptom.  Unless
> > there is something else involved.  Can you check whether this is the
> > staging/master combination that you are having problems with?
> >
> > > --snip--
> > >
> > > orking into jobs:  (1614 1613 1612 1611 1610 1609 1608 1607)
> > > logfile lilypond-multi-run-3.log (exit 1):
> > > Layout output to
> > > /tmp/build-lilypond-autobuild/out/lybook-db/57/lily-ffb6e70b-1.eps'...
> > > Converting to
> > > /tmp/build-lilypond-autobuild/out/lybook-db/57/lily-ffb6e70b-1.pdf'...
> > > Writing
> > > /tmp/build-lilypond-autobuild/out/lybook-db/57/lily-ffb6e70b-systems.texi... 
> > >
> > > Writing
> > > /tmp/build-lilypond-autobuild/out/lybook-db/57/lily-ffb6e70b-systems.tex... 
> > >
> > > Writing
> > > /tmp/build-lilypond-autobuild/out/lybook-db/57/lily-ffb6e70b-systems.count... 
> > >
> > > Processing 
> > > `/tmp/build-lilypond-autobuild/out/lybook-db/90/lily-68ca9147.ly'
> > > Parsing...
> > > Renaming input to: 
> > > `/tmp/lilypond-autobuild/input/regression/in-note.ly'
> > > Interpreting music...[8][16][24][32][40][48]
> > > Preprocessing graphical objects...Backtrace:
> > > In
> > > Exception during displaying of backtrace: wrong-type-arg
> > >
> > > /tmp/build-lilypond-autobuild/out/share/lilypond/current/scm/stencil.scm:56:7: 
> > >
> > > In procedure fold in expression (fold (lambda #
> > >   #) (car stils) ...):
> > > /tmp/build-lilypond-autobuild/out/share/lilypond/current/scm/stencil.scm:56:7: 
> > >
> > > Wrong type argument in position ~A (expecting ~
> > > A): ~S
> > That's stack-stencils-padding-list, used for the word-wrapping kinds of
> > markup commands.
> >
> > > Assuming the 'in-note.ly' is at fault here (this is what
> > > 90/lily-68ca9147.ly' is), this is
> > That does not make a whole lot of sense:
> >
> > > \sourcefilename "/tmp/lilypond-autobuild/input/regression/in-note.ly"
> > > \sourcefileline 0
> > > \version "2.17.6"
> > >
> > > \header {
> > >    texidoc = "LilyPond does in-notes.
> > > "
> > > }
> > And that's everything that _is_ in the snippet regarding the doc run!
> > The rest should have already complained during "make test":
> >
> > I seem to remember that we had the same problem (namely in
> > stack-stencils-padding-list) somewhat recently but don't remember what
> > it was exactly related to.
Just an FYI really. I kept getting this over and over yesterday no 
matter what I did - deleted my whole LILYPOND_GIT env and started again 
(downloaded it all from scratch), and I also restored from a known good 
backup (using LilyDev in a VM makes this very convenient).

But no matter what I did I always got this problem and couldn't seem to 
merge staging.

I did a few make and make doc, out-of-tree builds manually and they all 
went fine no problems.

Eventually I found the trigger - although I don't understand the cause.

As I use a VM I can set up my LilyDev env to use more or less the number 
of CPUs my physical server has. I usually use 7 (my Desktop here has 2 
quad cores) as I like to leave 1 for 'other stuff' I may be doing - as 
lilydev runs all the time I forget, and didn't want it interfering (too 
much) with day-to-day operations.

However it seems that for some reason (probably doing some comparative 
build tests) my patchy config was set to use 8 (make -j8 CPU_COUNT=8 
doc) so when I run the test-patchy scripts I was shaving about 2-3 
minutes off of make doc time, not a huge amount but when you're testing 
4 or 5 patches on the go and merging, it adds up.

However it seems that while test-patchy has no problems with this or 
rather the process that test-patchy does has no problem, patchy merge 
does. But only when using 8 CPU for the make options.

I tried with my VM set to only see 8 and then 7 CPUs but in both cases 
using make -j8 caused this odd error - in the same place every single 
time. That is even when the OS didn't even have 8 CPUs but make was told 
to use 8 if it could it failed. As soon as I moved down to -j7 with 
either 7 or 8 CPUs available to the OS for LilyDev it all worked.

I seem to recall that Phil said that if you only had X CPUs and used 
make with a -jY setting where X< Y it didn't seem to matter (at least to 
him) and that seems to be right.

So I did another test where my VM can only 'see' 6 CPUs but told Patchy 
to use 7 (make -j7) and this worked without problems. So I told patchy 
(still on my 6 CPU system) to use -j8 and it all fell over once more.

So I don't know if this is some esoteric make problem or a symptom of 
some other 'problem' within our own code while specifically building doc 
(as intimated above by David - I think).

Would it be possible for someone else to edit their 
.lilypond-patchy-config file to use the -j8 option (no matter how many 
CPUs they have) and see fi they get it too? It kicks in pretty near the 
start of the make doc so you wouldn't have to wait that long.

James

James