octave-bug-tracker
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

## [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

 From: Dan Sebald Subject: [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results Date: Sun, 26 Aug 2018 01:22:16 -0400 (EDT) User-agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0

```Follow-up Comment #10, bug #54567 (project octave):

KT, here's a test of the unsigned 64-bit integer using min/max functions:

octave:110> x = nth_element (x64, 2:3, 2);
octave:111> min (x, [], 2) + (max (x, [], 2) - min (x, [], 2)) / 2
ans =

9
12
18446744073709551608

[The following does not work generally]
octave:112> x(:,1) + (x(:,2) - x(:,1)) / 2
ans =

9
12
18446744073709551608

octave:113> x(:,2) + (x(:,1) - x(:,2)) / 2
ans =

11
14
18446744073709551610

I tested what happens when the unsigned "wrap" occurs.  Well, Octave is
programmed to not wrap.  I was curious what happens with wrapping because
sometimes one can devise a formula that allows integer wrapping so long as it
is guaranteed to unwrap.  Since Octave truncates rather than wraps, there's no
tricks that can be done.

OK, so let's think about an int64 formula then...

_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?54567>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

```

reply via email to

 [Prev in Thread] Current Thread [Next in Thread]