[PATCH] Notes (and possible fixes) for several small problems with MIDI output (issue #1661 among others)

[Heikki Tauriainen]

Hi,

For your information:

Here are some notes on a few minor defects that I have observed regarding
LilyPond's MIDI output together with links to patches with my attempts to
fix them.

I apologize for the long message, but since many of the issues share some
similarities, I considered it easiest to describe them together in a single
message.

About the patches: although I found these defects in the stable 2.14.2
release (and wrote the patches originally for this version), I modifed the
patches to apply cleanly to the 2.15.41 development release (which seems to
still have also same issues with MIDI output -- the source files affected
by my patches haven't really changed much since the 2.14.2 release).



====== ISSUE: instrument changes do not take effect as expected ======

Issue:

Changes in MIDI instruments sometimes take effect only after a note, not at
the note to which the change is already expected to apply.  From the user's
point of view, this problem occurs in an unpredictable way since even
seemingly irrelevant changes to the input file can make the problem
disappear.

(Note that this issue is very similar to known issue #1661
<http://code.google.com/p/lilypond/issues/detail?id=1661> which concerns
MIDI dynamics taking effect only after a note.  More on also this issue
below.)


Example:

------ Example 1 ------

S = {
  \key c \major
  \repeat unfold 2 { e'4 } e'2
}
A = { \repeat unfold 2 { e'4 } e'2 }
T = { \repeat unfold 2 { gis4 } gis2 }
B = { \repeat unfold 2 { e4 } e2 }

\book {
  \score {
    \new Staff = "Staff" \with { midiInstrument = #"choir aahs" } <<
      \S
      \A
      \T
      \B
    >>
    \midi { }
  }
}

------

On my Debian GNU/Linux system, both LilyPond 2.14.2 and LilyPond 2.15.41
(compiled from source tarballs with GCC 4.7.1) appear to consistently render
this input file into a MIDI file that, when played, does not sound right
(tested with timidity 2.13.2 and fluidsynth 1.1.5 + pmidi 1.6.0): more
precisely, the specified instrument takes effect only after the first chord
has already been played.  For example, when run on this MIDI file alone,
timidity 2.13.2 plays the first chord using acoustic grand piano as the MIDI
instrument.

Oddly, the problem disappears if, for example, any of the following changes
(which do not appear to have anything to do directly with MIDI instruments)
are made to the input file:

- remove the "\key c \major" line from the S block;
- change all \repeat unfold 2 's to \repeat unfold 1 's (or remove the
  \repeats altogether);
- change the Staff "Staff" to an "unnamed" one (\new Staff \with { ... } );
  or
- remove any one (or more) of \S, \A, \T or \B from within the Staff.


Analysis:

The MIDI track in the MIDI file gets built by a Midi_walker
(lily/midi-walker.cc), which translates Audio_items to Midi_items and
appends them to the MIDI track (Midi_walker::process,
Midi_walker::output_event, Midi_track::add in lily/midi-chunk.cc).  It seems
that the order in which Audio_items occurring "at the same time" get
processed depends on the structure of the input file in an unpredictable way
(at least from the user's point of view): when rendering the original
example, the Midi_walker sees an Audio_instrument event only after having
already processed some Audio_note events occurring at the same time, but
this order gets reversed if the input is modified (so that the
Audio_instrument event gets processed before any Audio_note events).  Since
Midi_walker::output_event appends converted events to the MIDI track in the
order in which they get processed, the result with the original example is a
MIDI track in which the first Midi_instrument event is already preceded by
"note on" events.  Although I am not familiar with the actual MIDI file and
track format specification as to what it might require about the way that
events occurring at the same time should be interpreted, to me it would at
least seem plausible that adding "note on" events to the track before the
instrument change event could produce exactly the kind of result that was
observed.


Proposed fix:

Ensure that, whenever there are multiple events occurring at the same time
to be added to a MIDI track, all MIDI "note on" events come only after all
other types of events (prioritizing all other event types over "note on"
events).  In the proposed patch, this is done by rewriting the
Midi_track::add function (lily/midi-chunk.cc) to search for the correct
insertion position for each new event to be added to a track.

Link to patch:

<http://koti.welho.com/htauriai/lilypond-patches/note-event-order.diff>

Affected file(s): lily/midi-chunk.cc



====== ISSUE #1661 (volume changes do not take effect as expected) ======

The above modification fixes the problem with Example 1.  Due to the
similarity of the instrument change problem (issue #1661), I tested whether
the fix might improve the behavior also in this case (since after
the fix, also all Midi_dynamic events ought to get placed before any "note
on" events occurring at the same time).  Here's the example from
issue #1661 (<http://code.google.com/p/lilypond/issues/detail?id=1661>):


------ Example 2 -----

% MIDI dynamics take effect one-note-late, if they are in a separate voice
% Below, we expect each measure to sound the same
\score {
  \new Staff  <<
      { c'2\ffff  d'\ppp  | c' d' }
      { s1 | s2\ffff s2\ppp }
    >>
  \layout {}
  \midi {}
}

------

(The behavior that I observe on my system with this example is that the
first c' and d' sound with \ffff and \ppp as expected, but the second
c' sounds with \ppp and the d' with \ffff.)


Analysis:

Unfortunately, the fix does not appear to help in this case.  Searching
further in the source code, it appears that this is because all Midi_dynamic
events get effectively ignored during output (from
Staff_performer::acknowledge_audio_element in lily/staff-performer.cc, I
understand that this has to do with the 2.14 change of encoding dynamics as
note velocities instead of MIDI volume).

However, I believe that Staff_performer::acknowledge_audio_element has a
similar defect as regards the output produced when the processing order of
Audio_items occurring at the same time varies.  More precisely, I believe
that, with Example 2, the following scenario is possible:

1. Suppose that, when processing "d'\ppp", the Staff_performer will first
   acknowledge an Audio_dynamic event (\ppp) and then an Audio_note event
   (c'2).  When processing the Audio_dynamic event, the Staff_performer's
   dynamic_map_ will be updated with the "current volume" of the current
   voice (\ppp); when processing the Audio_note event, this volume then gets
   applied to the note.

2. The next audio items to be processed are another pair of
   Audio_dynamic (from "s2\ffff") and Audio_note (from "c'") events
   occurring at the same time.  In this case the Audio_note event gets
   processed first: however, since dynamic_map_ still maps the "current
   volume" of the voice to \ppp, this volume gets applied to the note.
   When the Audio_dynamic event gets processed, this has the effect of
   only updating the "current volume" in dynamic_map_ to \ffff, but not
   the volume of the already processed simultaneously occurring c' note.

3. The same thing happens with the last pair of events: also in this
   case the Audio_note event corresponding to the final d' note gets
   processed first (applying the volume \ffff to this note from
   dynamic_map_, and updating dynamic_map_ only afterwards).


Proposed fix (based, hopefully, on a correct understanding of the  
calling conventions of Staff_performer callbacks as described in
<http://www.lilypond.org/doc/v2.15/Documentation/contributor/useful-methods-for-information-processing>):

Instead of adjusting volumes of individual notes as they are processed
(which could lead to the scenario described above), collect the notes played
in each voice at a single translation time step, and update the notes'
dynamics only in stop_translation_timestep to ensure that dynamic_map_ is
up-to-date w.r.t. any volume changes which might have occurred at this time
step.

(Of course, this will force all notes played in a single voice at a
translation time step to share the dynamics of whichever Audio_dynamic that
gets processed last in that time step, but since the order of processing the
events is not easily predictable from the input anyway, I would believe it
to be difficult to make constructs "equivalent" to

\new Voice { << { c'\ffff } { g'\pppp } >> c' }

produce the expected results in a consistent way without a mechanism to
specify the dynamics of individual notes as properties of the notes
themselves, or the processing order of simultaneous events explicitly in the
input -- this example also leaves it ambiguous as to what dynamics should be
applied to the final note.  To guard against unexpected surprises in MIDI,
the fix could probably be still improved with logic to display a warning if
the Staff_performer detects multiple dynamic events occurring in a single
voice at the same translation time step.


Link to patch:

<http://koti.welho.com/htauriai/lilypond-patches/staff-performer-dynamics.diff>

Affected file(s): lily/staff-performer.cc



====== ISSUE: "Initial" dynamics of a Voice may be out of range ======

Issue:

When using midiMinimumVolume/midiMaximumVolume (or a custom instrument
equalizer) to limit the volume of MIDI instruments, it is still possible to
end up outside the range in Voice contexts whose first dynamic marking is a
(de)crescendo instead of an explicit dynamic mark.

Example:

------ Example 3 ------

\book {
  \score {
    \new Staff \with {
      midiInstrument = #"oboe"
      midiMinimumVolume = #0.1
      midiMaximumVolume = #0.2
    } \new Voice {
      % These notes will sound louder than should be allowed.
      c'4_\< c'4 c'4 r4\!
      % This note sounds at the maximum allowed volume (scm/midi.scm)
      c'2._\sf r4
    }
    \midi { }
  }
}

------


Analysis:

The Dynamic_performer (lily/dynamic-performer.cc) class uses its
last_volume_ member variable to keep track of the current volume (an
absolute value between 0.0 and 1.0).  All explicit dynamic markings change
this current volume within the permitted absolute volume range (as
documented in the Notation Reference).  However, since the example does not
begin with an explicit dynamic marking, the volume at the beginning of the
crescendo will end up being the unequalized default value of last_volume_
(0.5, which is greater than the maximum allowed volume in this example).


Proposed fix:

Ensure that last_volume_ is equalized even if the first dynamic event in a
Voice context is a (de)crescendo instead of an explicit dynamic marking.
Also display a warning since using even the equalized value 0.5 is not
necessarily what the user expects since Dynamic_performers "do not get
inherited" from their enclosing contexts, as might be incorrectly expected
by looking only at the generated notation, in an example such as

\book {
  \score {
    \new Staff {
      \new Voice {
        c'2\sf
        <<
          \new Voice = "A" { \voiceOne g'^\sf g' g' }
          \new Voice = "B" { \voiceTwo c'_\> c' c'\ppppp }
        >>
      }
    }
    \layout { }
    \midi { }
  }
}

(Here, the first c' in \voiceTwo will already sound quieter than the first
note of the outermost Voice.  With the patch, a warning will be shown in
this case.)


Link to patch:

<http://koti.welho.com/htauriai/lilypond-patches/initial-volume.diff>

Affected file(s): lily/dynamic-performer.cc



====== PATCH: (de)crescendos terminated with \! ======

This is not really a defect, but just a minor update to the patch I sent to
the lilypond-user mailing list in last September
<http://lists.gnu.org/archive/html/lilypond-user/2011-09/msg00134.html>
concerning respecting MIDI volume ranges with (de)crescendos terminated with
\!.  On these (de)crescendos, LilyPond will try to decrease or increase the
current absolute volume by 0.25 (keeping the absolute volume always between
0.1 and 1.0, however); my previous fix just changed the the fixed limits 0.1
and 1.0 to the corresponding equalized volumes.

The patch below will change the attempted (fixed 0.25) change in the
absolute volume to 1/4 of the allowed volume range instead to reduce
the possibility of immediately hitting the minimum/maximum volume on
instruments given only a very narrow volume range.


Link to patch:

<http://koti.welho.com/htauriai/lilypond-patches/volume-range.diff>

Affected file(s): lily/audio-item.cc



====== DISCLAIMER ======

I consider myself just a LilyPond user who occasionally lurks on the mailing
lists (but who also happens to a coder, and hardly at all a musician).
Since there appears to be only infrequent discussion on open issues related
to MIDI output and their possible workarounds on the mailing lists -- I
guess simply because the vast majority of people use LilyPond mainly for
music typesetting -- I thought that I'd have the best chance of getting
rid of some minor annoyances in LilyPond's MIDI output if I tried to analyze
the problems and fix them myself.  Unfortunately, I'm only learning and
guessing things about the inner workings of LilyPond mostly as I try to read
and understand the source code, and I'm sure this lack of familiarity with
the organization of the code is reflected (badly) in the quality of the
patches -- for example, I wouldn't be at all surprised if some of my
patches to the C++ code could be implemented in a much more elegant way
with just a few lines of Scheme in some appropriate place :-) )  Anyway,
I decided to share my observations and patches in case they might still be
of help to improve the behavior of this excellent application.


--
Heikki Tauriainen