Re: how would one cross-reference two [or more] books?

[Jeff Olson]

(including lilypond-user)

On 1/12/2023 7:01 AM, Kieren MacMillan wrote:

> Hi all,
>
> A very large part of my composition [and thus Lilypond engraving] 
> career is concerned with very large, multi-format musical dramas 
> (operas, musicals, plays-with-music, dramatic song cycles, etc.). A 
> typical set of outputs might be:
>
> – full orchestra/band score
> – individual orchestra/band parts
> – Piano/Conductor score (reduction/arrangement)
> – Piano/Vocal score (a *different* reduction/arrangement)
> – Vocal/Choral score (reduction/arrangement)
> – libretto
>
> I would *love* to be able to have a note in the (e.g.) P/C score that 
> says “This song is found on pg. 12 of the Vocal/Choral score” — at the 
> very least at the song level, but optimally at any arbitrary point in 
> the score (e.g., top of every page, beginning of a given section of a 
> song, etc.).
>
> 1. What mechanism(s) would be best used to accomplish this?
>
> 2. Can it be done [relatively easily] if each of the outputs are 
> driven from an individual .ly file, or would it really require all 
> outputs to be coded in the same .ly file [with separate \book blocks]?
>
> Thanks!
> Kieren.
Hi Kieren,

IIUC, this sounds like the same problem as how to make an index of items 
across multiple books.

Back in 2.22 where I was running out of 32-bit memory, I was forced to 
try dividing my 500+ page compilation (logically one book) into multiple 
books in order stay under the memory limit.  I'm guessing the end of a 
\book throws out all of its grobs while keeping its stencils, and that 
releases a lot of memory.

But in my work, having an index at the end was absolutely essential, and 
it had to cover everything as though it had been multiple \bookparts 
instead of multiple \books.

The problem was that \page-refs into other \books always failed (they 
fell back to the default arg).

The solution was to save the \page-ref output as stencils at the end of 
each \book, i.e. to resolve the \labels while you still have them.

In my case, there was an index data structure that was designed to save 
\labels, which ordinarily would be converted into markup stencils when 
the index was being printed at the very end.

What I did was make a function that would re-scan that index structure, 
resolving any stored \labels into stencils, if that hadn't already been 
done (i.e. if they weren't already stencils, resolve them to stencils).  
When it was time to print the final index, you'd test the supposed 
\label values to see if they were already stencils: (if (ly:stencil? 
label) (markup #:stencil label) (markup #:page-ref label "XXX" "?"))).

Look in the attached dual-index-dev.ily (where dual meant something 
else: a sorted index plus a simultaneously kept unsorted index). See "% 
save-page-refs command" and "% index command" for the relevant code.

Where things got beyond my abilities was for the parts of document that 
were before my explicit \books.  This is demonstrated by the simple file 
MWEindex.ly.  It has two books, labeled No.1 and No.2., with the 
non-book stuff before them named "Top" and the stuff after named No.3.  
It makes a simple index showing those four items, along with their page 
numbers, which did get correctly resolved outside of the two books:

Exercise 0 . . . . . Top . . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . .Sav1
Exercise 1 . . . . . No.1 . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . 101
Exercise 2 . . . . . No.2 . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . 201
Exercise 3 . . . . . No.3 . . . . . . . . . . . . . . . . . . . . . . . 
. . . . . . . . . . . . . . . . . . . . 1

No.1 does start on page 101 and No.2 does start on 201, and No.3 is 
technically correct as page 1.  But I could not get items from Top to 
resolve also to page 1.  Instead, it is replaced by the default value 
supplied as the tag argument to save-page-ref, so I could tell which 
invocation of save-page-refs was clobbering things.

It might help to note which pdf outputs contain which parts:

    "Number 1" is in MWEindex.pdf
    "Number 2" is in MWEindex-1.pdf
    "Top Level", "Number 3" and the "Exercise Index" are in 
MWEindex-2.pdf (the last to print)

Apparently the page-refs from Top are indeed added to the index data 
first, even though that part of the document is the last to be printed. 
(Expected)

What was unexpected was that the \save-page-refs tagged "Sav0" was 
executed (or whatever the non-imperative term is) last.  This left the 
item labeled "Top" in the index where it was found during book 1 (by the 
save-page-refs tagged "Sav1") which couldn't resolve it because that 
label was not in book 1.

I didn't know how to overcome this peculiarity, other than by not 
indexing any items in the Top section.  And I lost interest when I 
learned that 2.23 was 64-bit and shouldn't need division into books to 
keep memory in bounds.  Turns out, ironically, that I am now having to 
add lots more bookparts so that garbage doesn't build up to where it 
triggers a third-party GC bug on Windows.  Still waiting for that fix 
RSN (2.24.1++?).

But if you only cross ref things that are in bona fide \books, this 
technique of resolving and saving \page-refs while still inside those 
\books should work.

Jeff

dual-index-dev.ily
Description: Text document

MWEindex.ly
Description: Text Data