Re: [Denemo-devel] GSoC proposal to improve MIDI and audio capabilities

[Richard Shann]

Hi Dominic,

Sorry not to reply earlier, but it seemed ok to wait until the situation
became clearer with the gsoc process... comments interspersed below:

On Mon, 2011-04-11 at 01:31 +0200, Dominic Sacré wrote:
> Hi Richard,
> 
> On Thursday 07 April 2011 18:45:58 Richard Shann wrote:
> > On Thu, 2011-04-07 at 15:07 +0200, Dominic Sacré wrote:
> > > I didn't mean to imply that Denemo is full of broken features, just
> > > that my proposal is mainly about restructuring and cleanup, not
> > > about removing things.
> > 
> > yes, I didn't take offence, I understood that, my question is would
> > the re-structuring you propose allow the adjustment of the MIDI
> > timing during feedback, as exemplified by the Tempo slider. That is
> > if you could describe the sequence of events that would happen when
> > someone moves the tempo slider during playback then I would be more
> > confident that we weren't getting ourselves into a trap.
> 
> I'm not sure if I can describe the sequence of events that would happen, 
> as I'm still struggling to understand how the current codebase works, or 
> more importantly why it was written the way it is.

Well, I can help with that, I think I have been over everything in
fluid.c, and it is worth noting that jackmidi.c is a clone (now somewhat
out of date) of the code in fluid.c (broadly speaking) - Jeremiah did
the cloning. (That is, I have developed fluid.c further since Jeremiah
last cloned the code into jackmidi.c).


> What I can tell you is that things like the tempo slider would just keep 
> working like they do now. With the changes I'm proposing, during 
> playback all MIDI events would still be read one by one from the SMF 
> structure (see fluidsynth_play_smf_event() and 
> jackmidi_play_smf_event()). The difference would just be that currently 
> it's up to FluidSynth to create its own output ports (or access the 
> soundcard, depending on the underlying audio API), while I'm proposing 
> to do that by ourselves in Denemo. Nevertheless, the way MIDI events are 
> fed from the SMF structure to the FluidSynth engine would remain more or 
> less the same.
> 
> That said, there's of course also the timing issue that Philipp 
> described. Now that I'm thinking about it, this is probably what 
> prompted the suggestion of "making midi/audio output realtime-capable" 
> as a GSoC project. Having written some JACK clients I automatically read 
> that as "make JACK support realtime-safe", but bad MIDI timing is 
> actually a different issue that has nothing to do with realtime-safety 
> per se.
Yes, it is just that once it is in a separate thread it can be given a
higher priority. (Well, that is my understanding, but I should say I
have never written a threaded program, only interrupt handlers)

> 
> As far as I can see, both the FluidSynth and JACK MIDI backends use 
> g_idle_add() to queue MIDI events for playback. It's pretty obvious that 
> timing-wise this is a terrible idea. Could anyone explain why the code 
> was written that way?

Well, you go on to explain why:), the effort of writing a proper
multi-threaded model. The thing you are probably missing is that written
like this it does what I need it to do - it allows me to listen to a
score with me stopping it when I hear a mistake. That is, as it is, it
provides a tool for "proof reading" scores by listening to them. I have
made it further use of this playback mechanism (e.g. the playalong
facility, where the score plays in response to you playing one of the
parts) and I am sure there is more to come. But I can believe a better
experience could be had if the playback timing were done differently.
And I know that there are other uses for which uninterrupted playback is
crucial, hence the project.


> A proper multi-threaded model seems to be the way to solve this, and 
> again, Philipp explained this much better than I did (in fact, I didn't 
> mention in at all...). I still believe the best approach is to tackle 
> all these problems at once in one major overhaul.
> 
> > > It has also been suggested that audio input is only semi-usable
> > 
> > I use it every week to tune my harpsichord. But I have not heard
> > reports of people using it for inputting notes to Denemo - it
> > certainly still does that - due to noises in the room I usually get
> > a few notes inserted before I hit the Tuning button - but it will
> > surely have some bit rot by now - it has lots of fancy pattern
> > following features ... (I got a MIDI keyboard a couple of years ago
> > and have never looked back).
> > 
> > > and
> > > should not be a high priority. I'm not so sure about that, and
> > > would rather fix it than remove it, as it's a pretty nifty
> > > feature. Incidentally, I haven't been able to test it yet. At the
> > > moment all my usable hardware audio inputs are JACK-only
> > > (firewire), and audio input in Denemo only works with PortAudio.
> > > Did I mention I was planning to fix that? ;)
> > 
> > yes you did, and I only mentioned that if you needed to cut corners
> > to achieve the goal audio input could be left untouched, it would
> > not break anything and would not be broken if it was left untouched.
> 
> Fortunately, removing the PortAudio dependency from the audio input code 
> doesn't look very difficult. It may even be easier to just do that, 
> rather than working around this part of the code just to be able to 
> leave it untouched, while the rest of the audio/MIDI I/O code is turned 
> upside down...

Since I wrote I have spooted an application for Audio in which I have
put on the todo list http://denemo.org/index.php/Audio_Score_Mixer so I
will be following developments here with interest:)

> 
> As for cutting corners, in the latest (final) version of my GSoC 
> proposal I de-emphasized support for Windows and OS X.
Most testing will get done by Windows users (though they are poor at
reporting) - if we stick to port(audio/midi) then we will get the
portablility I hope.

best,

Richard

>  I wouldn't 
> consider my work finished until those are supported (as they are now), 
> but building and testing Denemo on all these different platforms is a 
> can of worms I'd rather not open while I'm on such a tight schedule.
> 
> 
> Best regards,
> 
> Dominic