I'm not sure if it could be the issue or not. But I
remember a long while back (many years) FluidSynth used to
have issues with denormal numbers causing excessive CPU
usage. That would be floating point numbers which are very
small, but not 0. I don't know if ARM processors deal with
denormal numbers in software, which would result in poor
performance and possibly floating point exception handlers
being triggered. It seems like this would more often occur
when the audio went towards silence.
Some general information on this:
Might be worth looking in to. Code was added in certain
areas to detect if a number was below a very small value and
if it was, then it would be forced to 0. Maybe there are
still some areas where this is occurring which are only
affecting this particular platform?