Serious feedback and improvement headroom

[Urs Liska]

Hi devels,

I think I have some serious feedback to share, and it may lead to even 
more feedback and input.

This week I had the opportunity to present myself, LilyPond and my 
"plain text workflows" topic to a significant group of the Henle staff. 
In order not to whet your appetite too much: Currently there isn't a 
chance to get a foot into that business. They have very strict 
structures, and practically everything I would have to offer is very 
incompatible with these. So for them adopting LilyPond would mean 
throwing mostly everything out that they have built over the past 
decades. Nevertheless they were very interested and _did_ see the point 
in using version control for edition projects. Actually they even 
realized that this would vaporize a number of annoying issues they are 
facing every day ...

Apart from not having had an "accountable" success I had a paid trip to 
Munich, a few hours in Munich's sun - and I'm now proud owner of a 
beautiful engraving plate of m. 26-30 of the second movement of Chopin's 
f minor concerto :-)

###

The most interesting aspect of the meeting was that Henle's (only) 
in-house engraver was present too, and this may become a fruitful 
contact. He is using Amadeus, a Linux (!) program he bought for 20.000 
Euro in 1988 and which has been out of development for something like 15 
years now. IIUC they have two external teams of engravers, and those 
_do_ use Finale and Sibelius.

Amadeus is surprisingly similar to LilyPond in many respects, and as 
he's aware that he may not continue to use Amadeus forever LilyPond 
might be the obvious choice to switch to. However he had a number of 
questions and aspects where he considers Amadeus (still) significantly 
superior, and we should do good in trying to get as much input out of 
that contact as possible. I think _any_ comment from an engraver who's 
making scores for Henle day by day and uses a very similar tool to ours 
should be taken very seriously.

###

I got the Amadeus manual and will probably be able to share it with you, 
but I didn't get an answer so far.

First: Amadeus is a compiled system with an input language that is in 
parts extremely similar to LilyPond's while in others absolutely not.
Other than with LilyPond input is by design edited in different files 
that are processed with different editors and tools. I'd say you have 
separate files/editors for things like \header {} or \paper {} etc. and 
for the actual music. You edit these files with editors that (from the 
manual) resemble editors like emacs or vim - with lots of control 
commands. Particularly interesting is that (IISC) you write all layout 
tweaks in a separate file - exactly what we're working on with 
Jan-Peter's edition-engraver!
For viewing the results a shell script processes the files by 
preprocessing the parts, merging them to a preliminary format and 
finally producing an eps or pdf file. This looks like a traditional Unix 
pipe chain..

The music description itself is extremely similar to LilyPond, with a 
number of characteristic differences. I think that some of these are 
mainly a matter of taste or habit, some of them are (IMO) better in 
LilyPond (the handling of accidentals in the input for example: In 
Amadeus you write the "visible" notes, that is: the meaning of "d f a" 
depends on the currently active key signature). But some others look 
interesting, and I'll post them on bug-lilypond to discuss if some of 
them might warrant a feature request.
But beyond the basic music items there are significant conceptual 
differences in the input language. In general LilyPond's language is 
more verbose than Amadeus'. A few examples:

treble F 4/4 is the equivalent to

\clef treble
\key f \major
\time 4/4

(,9 .... ),10_a140140h30
seems to be an equivalent to a \shape invocation

Personally I think that LilyPond's approach is a very good compromise 
between Amadeus' assembler-like appearance and the confusing verbosity 
of e.g. XML formats. But this is something Henle's engraver would 
consider a significant problem because he thinks that he needs 
considerably less time entering the music in Amadeus.

This becomes particularly important when it comes to tweaking output. He 
wrote (my translation): "My first look at LilyPond [through my 
presentation and a follow-up email] shows similarities to Amadeus, but 
OTOH I have the suspicion that the operation isn't sufficiently mature 
to allow economic ans fast work. I think, you just have to input too 
much information to get an optimal result."
But he's also arguing at the level of actually entered number of 
characters. As a response to my argument that I prefer explicit pitches 
(a fis is a fis, regardless of needing a sharp or not, while in Amadeus 
a pitch is only defining a notehead position) he sais: "I don't 
understand why I should always enter the accidentals explicitly when 
they are already defined by the key. When I'm seeing the current key in 
the score display, I do know that in es major the a is actually an as, 
isn't it? Apart from that the editor always shows me the currently 
active key. The encoding in Amadeus is always unambiguous and can't 
produce misunderstandings. I have the impression that LilyPond always 
needs twice or three times the characters for the same content. That's 
offputting ..."
While I still think that explicit pitches are better for a number of 
reasons this _is_ the way someone thinks who has to produce 1.500-2.000 
pages of high-quality scores per year. For him each additional character 
makes a difference.

I don't suggest any significant changes in our input syntax. But I want 
to point out that editing efficiency on that level _is_ an issue we 
should keep taking into account when it comes to professional work. For 
this guy it makes a difference if he can (thousands of times) type "ho" 
instead of "\stemUp". And we all know that the process of tweaking 
output isn't that straightforward with LilyPond (although I very much 
appreciate all the little and bigger improvements we constantly see).

###

Another aspect that I found very astonishing is compilation speed. Of 
course Amadeus files have to be compiled before the result is visible, 
and this can be automatically done upon save. I think Frescobaldi's 
behaviour with Ctrl-M and by now the autocompile is an equivalent 
approach here. But the engraver claims that recompilation of a 30 page 
scores needs 2/10 of a second on an average computer. So this _is_ a 
fundamental difference, because he _does_ have practically instant 
WYSIWYG while still benefitting from text input and the compiled approach.
I don't have a clue as to how Amadeus achieves this. What comes to mind 
at first is a very smart approach as to what has to be recalculated and 
what not. As Amadeus also creates high-quality engraving I can't really 
imagine that this difference is only due to efficient programming. 
Although I can imagine that a program developed in the 1980s has a 
significantly stricter approach to runtime efficiency.
But I can also imagine that it has to do with the way C++ and Scheme 
code interact with LilyPond.
In another context I see a similar thing with LaTeX: Compiling a file 
with lualatex and fontspec takes longer by orders of magnitude than with 
plain latex. So maybe we really have a conceptual issue with the 
efficiency of LilyPond's runtime work.

As I can't imagine that we can speed up LilyPond's processing time by 
95% I really think we should find ways for partial recompilation. This 
is reportedly also the approach the new Steinberg application takes. For 
example when entering music I could do with only updating the currently 
interesting section (maybe a system would be a good unit for that).
Maybe it would be possible to create compilation modes that have a 
significantly sloppier approach to breaking. There are many stages in 
the development of a score where I don't care at all about the layout of 
the complete score. I recall that in Finale you could (probably: can) 
switch between score and roll mode (however this has been called). The 
latter was just one system without any breaks, and that is what can be 
used for music entry and for all kinds of _content_ editing. Of course 
this takes significantly less amounts of processing power, and it would 
be _much_ easier to create a way to only compile the currently changed 
measure (or maybe some more if there are living spanners).

Using strategies with skipTypesetting can help a lot in saving 
compilation time. But it would be an extremely useful enhancement if we 
could get some kind of partial recompilation _within_ the context of an 
existing score.
[I just got another email indicating that Amadeus will by default 
recalculate the current page only, and this works in near instant time. 
I think with this we can get to an area where calculating efficiency can 
make such a difference.]
What I would vote for very much is an option to recalculate the current 
system only, but with outputting the complete score anyway (however it 
may be determined what "current" is). This should work in many cases 
when a modification doesn't change line breaking. And it would 
dramatically increase the editing experience.

###
###

This email contains a lot of stuff, and I expect it to raise 
considerable discussion. Therefore I suggest to try keeping it organized 
and updating message titles as soon as we're discussing individual 
topics. I also suggest to respond to different topics separately, with 
creating recognizable subthreads. Otherwise it may soon become 
impossible to find thoughts you recall having read about ;-)

Best
Urs