Making a Score-Following DVD with Lilypond

[Silas S . Brown]

Imaginary poster wrote:
> this is a fake quote to get my post past gmane.  For some
> reason it keeps accusing me of "top posting" when I'm
> not, so I'm putting this in to try to persuade it to let
> me through.

(I think it's because this email contains snippets of Python
code, which have lines ending in colon that gmane thinks
are attributions.)

Here is a mini-howto for using Lilypond to make
a score-following DVD.  Any comments? / do you
want to put this up somewhere?

(NB this contains code snippets, and as I'm posting
through the GMane interface,
I do hope that it won't munge the code.)

Making a Score-Following DVD with Lilypond

The idea is to make a DVD of a score that will
be displayed on the screen as the music is being
played, and the score follows the performance.

You need a working Lilypond installation, and a
score in Lilypond (download one from
mutopiaproject.org for example).  Check that the
score will typeset without errors to begin with.
You will also need MEncoder, netpbm,
imagemagick, and a Unix-like environment (Linux,
Cygwin, etc).

It is a good idea to make a temporary directory
and put the Lilypond file(s) in it.  The process
below will make lots of temporary files and it
will help if they are in their own directory.
Also, it is possible that the presence of
unrelated files could confuse some of the shell
commands below if you are not working in a
dedicated directory.

Edit the .ly file (the main one if there are
several).  At the very beginning (right after
the \version if there is one), add this line:

  #(ly:set-option 'clip-systems)

Now find the \score block, and at the end of it
add the following:

\layout { clip-regions = #(list
          (cons
           (make-rhythmic-location 0 0 1)
           (make-rhythmic-location 99999 0 1))
        )}

Run Lilypond on the file.  If it is large
(symphony etc) then you may need a great deal of
virtual memory and time.  It will produce one
.eps and .pdf file for each system.  If it
raises a GUILE error after doing so, that's no
big deal if the .eps files have already been
written.

Now run the following shell commands:

mkdir tmp
for N in *-clip-*.eps; do mv $N tmp/$(echo $N|sed \
-e 's/.*-clip-//');done
mv *-clip.eps tmp/0.eps
cd tmp
export Sys=1
for N in $(ls --color=none|sort -n -r); do
  pstopnm -stdout -xmax=7200 -ymax=5760 -portrait $N |
 pnmscale -reduce 10 |
pnmtopng > ../sys$(printf %04d $Sys).png
  export Sys=$(echo $Sys+1|bc)
done
cd .. ; rm -r tmp

You now have the score's systems in
sys0001.png, sys0002.png, etc., in a format
that will fit on PAL DVD video.  Note that you
can also put non-Lilypond scores into this
format; for example you can take a scan of an
old printed score and manually break it up into
systems, saving each one as an EPS file before
running the above commands (if you do this then
you may like to try the "de-rotate" and "save area"
utilities at http://people.pwf.cam.ac.uk/ssb22/notes/
but remember to convert to EPS).  The advantages
of using Lilypond are that you don't have to
make sure the scan is straight and clean, you
don't have to manually cut out all the systems,
and the typesetting will probably be clearer
(unless you have a high-quality hand-typeset score).

The next step is to work out the
panning.  First make a list of all the systems:

  echo sys*.png > list.txt

Then, if there are any repeats in the piece, you
need to edit list.txt and make sure that the
repeats go in.  For example, if systems 1
through 27 are repeated, then you need to make
sure that sys0001.png through sys0027.png is in
list.txt twice before sys0028.png.  If necessary
you can preview the PNG files by doing

  for N in sys*.png; do display $N; done

(press Q to quit the display program and move to
the next system)

When your list.txt is ready, you now need to
play the audio and get timing information for
each system.  I don't suggest getting Lilypond
to work out the timing automatically, because
you will likely want the sound track to be a
live performance, which will probably not keep a
precise tempo.  Create a file get-timing.py with
the following contents:

import os,time
w=open("timings.txt","w")
t=time.time()
for s in open("list.txt").read().split():
  os.system("display -geometry +0+0 -immutable "+s)
  w.write(s+" "+str(time.time()-t)+"\n")

Back at the shell prompt, type:

python get-timing.py

but do not press Enter yet.  Get the sound track
(CD or whatever) ready to start; check that you
know how to start it instantly.  Then, press
Enter on the above command while simultaneously
starting the sound track.  You should see the
first system displayed.  When the performance
reaches the end of the first system, at that
exact moment press Q, and you should see the
second system appear.  Continue like this all
the way through the piece.  Finally, when the
last system displays, wait until the music
finishes and then press Q.

The timing information should now have been
logged to the file timings.txt.  If you are
happy with it, you can now generate the frames
of the video.  Copy the code below (between the
two "cut here" lines) into frames.py :

(I did have two "cut here" lines around this code, but GMane wouldn't let me
post it because it thought I was "top posting".  So now you have to work out
where the code starts and ends.)
(Also I had to munge the formatting a bit for the 80-character limit)

import os

screen_width, screen_height = 720, 576  # PAL DVD

sys_endtimes = [(l.split()[0],float(l.split()[1])) for l in \
filter(lambda x:x,open("timings.txt").read().split("\n"))]

def cut_images(itemNo, startX, out_png_filename):
    # cut the tapestry such that itemNo starts at startX
    while itemNo and startX>0:
        itemNo -= 1
        startX -= getWidth(sys_endtimes[itemNo][0])
    paste=[]
    while startX<screen_width and \
itemNo<len(sys_endtimes):
        left=-min(startX,0)
        width=getWidth(sys_endtimes[itemNo][0])
        if left<width:
            os.system("pngtopnm "+\
            sys_endtimes[itemNo][0]+"|pnmcut "+\
            str(int(left))+" 0 "+\
            str(int(min(width-left,screen_width-startX)))+" "+\
            str(getHeight(sys_endtimes[itemNo][0]))+\
            " >tmp"+str(len(paste))+".pnm")
            paste.append("pnmpaste tmp"+\
            str(len(paste))+\
            ".pnm"+str(max(0,startX))+" 0")
        startX += width ; itemNo += 1
    os.popen("|".join(paste)+"|pnmtopng>"+\
    out_png_filename,"wb").write(\
    blank_pnm(screen_width,screen_height))

width_cache = {} ; height_cache = {}
def getWidth(filename):
    if not width_cache.has_key(filename):
width_cache[filename]=\
int(os.popen("pngtopnm "+filename+\
" 2>/dev/null|pnmfile|sed -e 's/ by .*//' -e 's/.* //'").read())
    return width_cache[filename]
def getHeight(filename):
    if not height_cache.has_key(filename):
      height_cache[filename]=int\
(os.popen("pngtopnm "+filename+\
" 2>/dev/null|pnmfile|sed -e 's/.* by //' -e 's/ .*//'").read())
    return height_cache[filename]

def blank_pnm(width,height): return \
"P5\n"+str(width)+" "+\
str(height)+"\n255\n"+chr(0)*width*height

frameNo = 0
for i in range(len(sys_endtimes)):
    # scroll i from 0 to -width
    # (but add 100 pixels for error)
    frames = int(sys_endtimes[i][1]*25)-frameNo
    for f in range(frames):
       
cut_images(i,int(100-getWidth(sys_endtimes[i][0])\
*1.0*f/frames),"frame%09d.png" % frameNo)
        frameNo += 1

Now run:   python frames.py

It will write each frame of the video to a
separate PNG file.  This typically takes about
300-400 megabytes of disk space for every 50
pages of orchestral score, so if disk space is
limited then do one movement at a time.  Note
that the program will likely take many hours to
run.

Then to encode all the frames into an MPEG file
together with the sound track, type this:

mencoder mf://frame*.png -mf w=720:h=576:fps=25:type=png \
-oac lavc -ovc lavc -of mpeg -mpegopts format=dvd \
-vf scale=720:576,harddup -srate 48000 -af \
lavcresample=48000 -lavcopts
vcodec=mpeg2video:vrc_buf_size=1835:\
vrc_maxrate=9800:vbitrate=5000:keyint=15:aspect=16/9
-ofps 25 \
-o score.mpg -audiofile cdda.wav

where cdda.wav is a wav (or mp3) file of the
soundtrack (will likely be called cdda.wav if
taken from a CD using cdparanoia), and score.mpg
is the name of the destination file, which will
be an MPG file suitable for putting onto a DVD
using 'dvdauthor', 'dvdflick' or a similar package.


-- 
Silas S Brown http://people.pwf.cam.ac.uk/ssb22