FYI: encoding issues

[Han-Wen Nienhuys]

Hi all,


As you might know, the support for languages that use non-ASCII glyphs
(both european characters with accents and non-european characters) is
flaky: choices for glyphs are dependent on the -rather limited- TeX EC
font of the.  The problem of font encoding also bites us in the GNOME
backend, which some trickery to make Pango display the glyphs on the
GNOME canvas.

At the LDM 2004 (LilyPond Developer Meeting 2004. Basically this is
Han-Wen sitting with Werner in a bar in Oberhausen ), Werner Lemberg
and I have determined a course of action for LilyPond font handling.

Doing all this will be a lot of work, but I think the result will be
worth it: we will have

  * good support for non-western lyrics,

  * usable PostScript output  

  * better support for TeX's text formatting features.

  * A cleaner system for reading and processing fonts.


Here goes:

* Input encoding:

 - Use UTF-8 as input encoding.

   Encoding is a mapping between numbers and glyph-names.  Unicode is
   the most definitive standard, a system where each glyph is a 21-bit
   word.. The most convenient way to pack these 21-bits in a file is
   UTF-8, which is backward compatible with ASCII.

 - Files in other encodings

   (like latin1, or latin2) should be preprocessed (eg. with iconv) to
   produce UTF-8.

 - The current feature, where you can switch encodings on the fly,
   using \encoding should die, as there are no text editors that
   support this kind of switching.

* Make a distinction between two types of strings:

  - internal strings (for use in Scheme) remain in UTF-8

  - text to be printed on the page will processed as a list of
    glyph-names.

* Unify the current font into OpenType:

  - dynamics, numbers, modern notes, ancient notes should be merged
  into an OpenType font.

  [What about the brace font?]

  - This merging is done by pasting together pfas using FontForge, and
    writing the result as an OT font.

  - font parameters (staff-space, line thickness, stem attachment
    points) are stored in the OT font, since  OT fonts support
    arbitrary tables.

  - FontForge will create a default PUA map

   Unicode is a standard, but all music glyphs fall outside the
  standard (and will continue to do so), so the lilypond font must
  mapped in the PUA (Private Unicode Area).

   - Printing PS is done by embedding a CFF font. (CFF is a wrapper
   format for OpenType). 

   - The TeX backend continues to use the current jumble of PFA and
   TFM fonts.

   - The AFM fonts can be junked.

* Use FreeType to read the OT font, and extract metrics and tables

* Use Pango to render non-music texts.

  - Input to Pango is a list of glyphs. The output of Pango is a list
  of glyphs with (X,Y) positions.

  - If Pango is not there, dump glyph list directly to the output,
   unchanged.

  - Pango deals with language settings, ligatures, accents and
  kerning.

* The TeX backend will go back to basics:

  - with --format TeX, the input strings are dumped verbatim to the
  output .tex file.

  - we wil dump all text-strings to a separate file, to be processed
  by TeX to compute box extents

  - in a second run, TeX extents are used to determine extents of
  texts.

* Finally, we have to find a good default font for text. The EC font
  doesn't cut it

  - its quality is not high,

  - it has a limited set of glyphs

  - it doesn't look like traditional titling.



Need to check:

 * FontForge:

   - can we write CFF?

   - can we create spacing tables within an OT font?

 * Pango: does Pango work correctly with the PUA map in an OT font?
  Can we have glyph-list + (X,Y) output (so: can we use Pango without
  linking X11 in?)

 * Does pango + freetype work ok on cygwin? 



Other issues that Werner brought to my attention:

* We sorely need a system divider for orchestral scores.  Currently,
LilyPond orchestral scores are barely readable.

* There is a need for an \aftergrace music function, eg.

  \aftergrace { d1 }  { c16[ d] } c4

=>

  << { d1 }
     { s1*7/8 \grace { c16[ d] } } >> c4    

* There should be a way to make crescendi run up to (but not
  including) the next note; typically the crescendo ends at the
  barline before the next note. Preferably there should also be
  mechanism to have a crescendo over just a single note, ending it at
  the next note.

* There should be a separate clef for cue notes, a cue clef

* There should be a way to ensure that texts don't run outside the
  lines, preferably by forbidding certain breakpoints.

* Cluster notation: the odd notation with split stems.

* A far away wish: support skyline compositing of Stencils. 

-- 

 Han-Wen Nienhuys   |   address@hidden   |   http://www.xs4all.nl/~hanwen