Compression penalties

[Joe Neeman]

I want to replace the old page breaker and line breaker with the new
ones. The new algorithms are now (I hope) sufficiently powerful that the
old ones are simply special cases of the new ones, so this should be
do-able. But there is a problem.

With the new page breaking stuff introducing more constraints into the
page-breaking problem, we tend to get slightly worse horizontal spacing
than before (this much is unavoidable). The problem is that we tend to
get excessive compression which looks really bad. To fix this, I added a
(convex) compression penalty to the horizontal spacing problem.

The file simple-spacer.cc contains the following comment:
/*
    We used to have a penalty for compression, no matter what, but that
    fucked up wtk1-fugue2 (taking 3 full pages.)
 */

The good news is that this is no longer the case. When using the
ly:optimal-breaking page breaking algorithm _with_ horizontal
compression penalties, the algorithm chooses the right trade-off between
horizontal and vertical spacing and wtk1-fugue2 still fills 2 pages
nicely.

The problem is that we break baerenreiter-sarabande.ly. With compression
penalties, it goes from 6 lines to 7 (since ragged-bottom is true,
page-breaking is having no effect here). My preferred solution is just
to set system-count to 6 and forget about it.

I realise that part of the purpose of baerenreiter-sarabande is to test
the line-breaking algorithm and therefore I'm a bit wary of adding an
extra tweak to compensate for line-breaking changes. However: the number
of systems in the Baerenreiter original is not necessarily the optimal
number of systems for the horizontal spacing; it is the optimal number
of systems for the combined horizontal and vertical spacing. I therefore
think it's OK to force the number of systems, given that we've taken it
out of context of the vertical spacing.

What do other people think?

PS: Sorry for such a long email. It's just that I'm proposing to
intentionally break something and I want to see if it's OK.