Re: Sponsoring lilypond development Was Re: Score parts: instrument and duration

[Trevor Baca]

On 8/18/05, Han-Wen Nienhuys <address@hidden> wrote:
> Trevor Baca wrote:
> > Just mail to address@hidden ; the list is run by Gordon Callon in
> > Canada. Quite unfortunately there are no list archives :-( but, quite
> > happily, the list is responsive, open and professional (much like our
> > own, really).
> 
> Is this a conscious decision? It's easy to have a list archived
> (including spam-protecting addresses) through mail-archive.com or gmane.org.

I'm sure it's not conscious; SCORE's mailing list is run out of the
generosity of Gordon's time and I'm sure it's probably a simple matter
of Gordon not yet having hand the time to take advantage of what gmane
has to offer.

> >>>We could steal the users directly from there... But for this to
> >>>happen, we would have to reverse engineer the SCORE format.
> >>
> >>hum, it's a binary format right? I've forgotten that.
> 
> The point of SCORE is that everything can be very easily tweaked
> heavily, which is excellent for professional engravers. All those tweaks
> get lost if the .pmx is imported. Since the majority of time in SCORE is
> spent tweaking, I don't think importers matter much.

I tend to agree.

As a sidenote, it's always struck me that it's seems *much* easier to
lure users away from one music notation package and towards another
than it is to, say, lure users away from one word processor or
spreadsheet and onto another. Why? Almost certainly because notation
always has been and still is one of the most solitary,
non-collaborative exercises ever, from the 9th century right down to
the present. The vast majority of stuff typed up in MS Word and Excel
is meant to be mailed over to someone else's computer somewhere else
in your office (and likely modified). This just simply isn't the case
for the notation files we're all typing up: the final destination is
paper and we're probably not shipping notation sourcefiles anywhere,
ever, or at least certainly not to the extent that office documents
get kicked around.

Enter, then, the fact that Sibelius has been able to court away so
many Finale users (hundreds of thousands now, in fact): the beginning
of each new project is a chance for a user of a notation program to
leave and try out a new package.
 
> I think there is a one big problem with going to LilyPond for the
> majority of SCORE users. Adding a tweak is clumsy, and you have to rerun
> the program to see the result. Also the result of a tweak is less
> predictable, as anything besides extra-offset might impact spacing, line
> breaking, etc.  Also, working with multi-page documents is a nightmare,
> since page breaks are hard to get right.

This is the crucial question, right here -- the question of tweaking.
(Much less so the question of rerunning the program to see the result,
imo: you spend a *LOT* of time rerunning things in SCORE ... exactly
like in Lily, outputting to postscript is a completely separate
process than editing input ... you call entirely separate DOS
programs, in fact: SCOR4 to summon and use the editor and SCORLAS to
set up printing options and actually generate the postscript. So no
problems there for SCORE users interested in the LilyPond work-cycle.)

But the tweaking question really is important. And, actually, I think
H-W's assessment of Lily's tweaking affordances may be too dour, but
I'm not expert enough in both programs to be sure.

I see one huge primary technical difference as regards the tweaking
model, however: SCORE renders the *horiztonal position* of absolutely
every item on the page completely visible to the user; SCORE's own
language every visible item (clef, staff, note, slur, whatever) is
"code" (and there are 18 or so different codes, depending on how you
count) and each "code" comprises between 4 and 20 different
"parameters" numbered p1, p2, p3 ... as necessary. Some "parameters"
differ between "codes", but absolutely ALL "codes" have as their "p3"
the horizontal position on the page. The values for p3 are a unit-less
scale ranging from 0 to 200, where 0 marks left staff edge and 200
marks right staff edge; a p3 value of 100, therefore, marks the dead
center of the staff.

Spanning items (ties, slurs, brackets, ottava lines, same concepts as
in Lily), then, usually have two horizontal parameters, usually p3 for
the starting horizontal and p6 for the stopping horizontal; p4 & p5
usually then represent the starting vertical and stopping vertical,
respectively, for such spanners.

If we stop and think what this implies, for a moment, it means that,
among other things, it's entirely possible to rewrite the entire
spacing algorithm in SCORE as a *postprocessing* program that simply
looks at a SCORE .mus or .pmx file, finds the p3 (and possibly p6)
values for each "code" and does some math. If this sounds unlikely you
might be surprised that this is exactly what one member of the SCORE
community did: his results are superior to SCORE own spacing routine
(named "lj" for line-up and justify) and he licenses the result for a
reasonable sum to many other engravers in the community!

So, although SCORE's sources are quite closed the availabiliy of the
open .pmx format (from which the .mus binary format isn't actually all
that hard to reverse engineer) has created something of an open space
in which significant contributions are possible.

Anyway, the availability of *absolute* horizontal positioning of each
"code" on the page in SCORE is a major differentiator, imo, beween
SCORE and all other notation packages (and, unfortunately, in this
case Lily is actually in the Finale / Sibelius camp: horizontal
positioning of items is all relative, rather than absolute). Alignment
in SCORE is mostly rock-solid: want the start of your cresendo to
align directly with a notehead, or want an initial tempo indication to
align exactly with your time signature? Simply give them both the same
p3 horizontal position value (then picking whether things left- or
right-align is usually done by toggling some higher number p-value
between -1, 0, 1.)

Which brings up this structural difference in both the program design
and the user work-cycle interaction between SCORE and Lily:

The SCORE work-cycle is essentially:

1. cleartext input (editor or file)
2. interpretation of input by SCORE to generate .pmx or .mus
3. cleartext .pmx (or binary .mus) representation of the position of
every interpreted item in the score
4. interpretation of interpreted input by SCORLAS4 to generate postscript
5. postscript

The Lily work-cycle is then:

1. cleartext input
2. interpretation of input by Lily
3. interpretation of interpreted input to generate postscript by Lily
4. postscript

In other words, the .pmx / .mus stands as an *intermediate* artifact
in the middle of the pathway from input to postscript (which is
unavailable under the Lily work-cycle).

Unless I'm misreading (which I very well might be), Lily's code isn't
set up to afford the caching of the intermediate stages between input
interpretation and postscript output; if it were then that would open
up an entire editing stage for Lily users in place of sprinkling
extra-offsets in the initial input-entry stage!

> Practically speaking, I think Lily can only practically replace SCORE
> for single staff pieces. And then the placement of hairpins should
> definitely be fixed.

Well, hairpins should acquire a better interface (not just attaching
to rhythmic positions but also to "typographical" positions, like just
one side or the other of a barline, for example, or the start or stop
of entire measures) but that can be run as sponsored work, I'm sure.

But I disagree that Lily can only replace SCORE for single staff
pieces; why should this be the case? If Lilly's not there yet then
it's very close. What's actually missing? Possibly ...

1. (Horizontal) spacing regions throughout a piece

2. (Vertical) spacing regions throughout a piece (so that the
forced-distance between piano staves can vary from system-to-system);
there is the bleeding-edge example in the tips 'n tricks section, but
it's not end-user-ready

3. Either absolute horizontal positions open to the user (as mentioned
above) and / or glyph alignment to non-rhythmic points on the staff
(intial indications should be alignable to left time sig edge, right
time sig edge, left key sig edge, right key sig edge, etc; hairpin
start- and stop-points should be alignable to barlines or left text
edge, right text edge, etc); again, maybe these abilities exist
already, perhaps by examining position attributes of parent objects in
attachment, but if so I haven't yet figured it out yet

4. Flexibility in the slurring interface (explicity instructions to
the interpreter to say things like "slur starts notehead center; slur
starts left notehead edge; slur starts right notehead edge; slur
starts at stem tip; etc); FAIK those affordances may, again, already
exist in the current slurring interface somewhere in slur-details

Anyway, these changes should all be possible as sponsored work, it
seems to me, like everything else, and get us much closer to working
with the SCORE folks. I think we're closer than we think.

Trevor.