[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

[Kai Torben Ohlhus]

Follow-up Comment #5, bug #54567 (project octave):

Good catch with the (u)int64 data type.

I think the second version of comment #2 will deal with averaging two numbers
for the even case with (u)int8-32, single, double, and logical.  For (u)int64,
we might have to introduce an exception


## Perform the averaging in double precision to avoid intermediate
## overflows of finite data types like "int8".  For "int64" and
## "uint64" double precision is insufficient and we have to
## perform the division by two beforehand.

x = nth_element (x, k:k+1, dim);
if ((isa (x, "int64") || isa (x, "uint64"))
    && (any (x >= flintmax () / 2)))
  warning ("median: inaccurate result due to large integer values.")
  retval = sum (x / 2, dim);
else
  retval = cast (sum (x, dim) / 2, class (x));
end


The formula of comment #3 and comment #4 is hard to implement regarding
multi-dimensional arrays.  If you found a more elegant way, I would be very
happy ;-)

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54567>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/