[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

From: Kai Torben Ohlhus
Subject: [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results
Date: Sat, 25 Aug 2018 08:12:20 -0400 (EDT)
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36

Follow-up Comment #5, bug #54567 (project octave):

Good catch with the (u)int64 data type.

I think the second version of comment #2 will deal with averaging two numbers
for the even case with (u)int8-32, single, double, and logical.  For (u)int64,
we might have to introduce an exception

## Perform the averaging in double precision to avoid intermediate
## overflows of finite data types like "int8".  For "int64" and
## "uint64" double precision is insufficient and we have to
## perform the division by two beforehand.

x = nth_element (x, k:k+1, dim);
if ((isa (x, "int64") || isa (x, "uint64"))
    && (any (x >= flintmax () / 2)))
  warning ("median: inaccurate result due to large integer values.")
  retval = sum (x / 2, dim);
  retval = cast (sum (x, dim) / 2, class (x));

The formula of comment #3 and comment #4 is hard to implement regarding
multi-dimensional arrays.  If you found a more elegant way, I would be very
happy ;-)


Reply to this item at:


  Message sent via Savannah

reply via email to

[Prev in Thread] Current Thread [Next in Thread]