Some bugs (???) in lexer.ll, parser.yy of Lilypond 2.9.9

[Angelo Contardi]

Hello Han-Wen,

i'm not a musician (though i like music) but a C programmer with some experience
in flex/yacc parser (just C, not C++).
During this week i have take a look at lexer.ll (mostly) and parser.yy (just few
time) and i think there are some errors:

lexer.ll:
---------

1) The TEX syntax include both PUNCT and ACCENT so in the LYRICS may
   include ' and ` both "escaped" (ACCENT) and "non escaped" (PUNCT).
   To solve this problem probably you should define:

      PUNCT [?!:]
      
   And allow only escaped version of ` and ', like in C strings.
   Note that PUNCT and ACCENT appear only in TEX, that appears only in LYRICS.
   
2) Not a bug, but a possible simplification. The "semantic actions" in the
   status <quote> and <lyric_quote> are the same, except for the token
   returned (respectively STRING and LYRICS_STRING). So both status may be
   re-wrote in a compact form:
   
   <quote,lyric_quote>{             // modified line
      \\{ESCAPED}    {
         *yylval.string += to_string (escaped_char (YYText ()[1]));
      }
      [^\\"]+        {
         *yylval.string += YYText ();
      }
      \"             {
         yy_pop_state ();
         /* yylval is union. Must remember STRING before setting SCM*/
         string *sp = yylval.string;
         yylval.scm = scm_makfrom0str (sp->c_str ());
         delete sp;
         if (YY_START==quote)       // modified line
            return STRING;          // added    line
         else                       // added    line
            return LYRICS_STRING;   // added    line
      }
      .              {
         *yylval.string += YYText ();
      }
   }
   
   But my question is: do you really need to distinguish between STRING and
   LYRICS_STRING? If no, you may take in count a simplification of lexer
   (just one status for all "quoted strings") and parser.
   
   In any case you can "group" all kind of "start quote" in this compact way:
   
   #define start_quote()              \
      if (YY_START==lyric)            \ // start_lyric_quote() was called Just
         yy_push_state (lyric_quote); \ // in <lyric> status
      else                            \
         yy_push_state (quote);       \
      yylval.string = new string
      
   The "#define start_lyric_quote() ..." should be eliminated, and you can group
   ALL "quoted strings starting points" as follow:
   
   <INITIAL,notes,figures,lyric,chords,markup>{ // Have i forgot some status ?
      \"             {
         start_quote ();
      }
   }
   
   Note: bacause of in status <sourcefilename> and <version> "quoted strings"
         are resolved locally, they may include any char different from ".
         Is it OK or should be added two more "macro" VERSION_STRING and
         FILENAME_STRING to restrict the strings values.

parser.yy:
----------

1) %token BOOK "\book" should be probably "\\book" :-)

2) The "unary" rule:

   toplevel_music:
      Composite_music { }
   ;
   
   may be eliminated replacing "Composite_music { }" in toplevel_expression
   rule.

3) The grammar rules:

   chord_open:  '<' ;
   chord_close: '>' ;
   simul_open:  DOUBLE_ANGLE_OPEN  ;
   simul_close: DOUBLE_ANGLE_CLOSE ;
   
   May be probably replaced by the homonym (UPPER CASE) "Byson tokens":
   
   %token CHORD_OPEN  "<"
   %token CHORD_CLOSE ">"

   %token SIMUL_OPEN  "<<"   // DOUBLE_ANGLE_OPEN,  renamed
   %token SIMUL_CLOSE "\\"   // DOUBLE_ANGLE_CLOSE, renamed