octave-bug-tracker
[Top][All Lists]

## [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

 From: Dan Sebald Subject: [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results Date: Sat, 25 Aug 2018 12:30:26 -0400 (EDT) User-agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0

```Follow-up Comment #7, bug #54567 (project octave):

Nice article on mean computation...  This is why I wondered if it were better
to put this case of 64-bit average in the mean() script and call mean() inside
median() instead.  Could there be an internal library routine that is stable
that making a built-in mean() makes sense?

Still, here for median() it is only two numbers we are averaging, plus the
formula in the reference pertains to float values, not integer "wrap" effects.
Consider that the formula from the reference becomes

mu_2 = mu_1 + 1/2 (x_2 - mu_1)
= x_1  + 1/2 (x_2 - x_1)

which is the same formula as in Comment #4.

KT, keep thinking about the formulas; it shouldn't be too difficult to
implement conditionals with scripts.  They slow things down, but I'd prefer
being correct than have a warning message.  For example, the formula of
Comment #4 could be implemented as

retval = min (x, dim) + (max (x, dim) - min (x, dim))/2;

Maybe there is a way to generalize the formula from the reference for
integers.  The issue with integers is the loss of the fractional portion when
doing the intermediate mean in the recursion.  Perhaps there is a way to
accumulate those lost bits as some sort of residual and then at the very end
of the computation adjust the final computation with the residual divided by
N.

_______________________________________________________

<http://savannah.gnu.org/bugs/?54567>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

```