[Octave-bug-tracker] [bug #63089] dec2bin gives wrong answer for negative int16

[Arun Giridhar]

Follow-up Comment #5, bug #63089 (project octave):

I am attaching a WIP patch for testing. It is vectorized (no loops) and can
accommodate negative numbers below flintmax to the full range of int64 for
signed and uint64 for unsigned. It passes all BISTs except for these:


dec2bin (-2^31 -1)
dec2bin (-2^52)
error <negative inputs cannot be less than> dec2bin (- flintmax ())


In the first two cases, the expected result was a 53-bit string but the
patched version returns a 64-bit string with more leading 1s, which on
reflection I think is an improvement since the other conversions of negative
numbers when length is not specified return 8, 16, or 32 bits, so returning a
bit string with 64 bits is more consistent than 53 bits.

The last failing BIST is because the patched version *can* handle negative
numbers below flintmax, so the expected error does not happen.

The trimming-to-length part is subjective for negative numbers, because one
can either return the minimum number of bits plus a leading "1", or one can
return a fixed power-of-2 number of bits with several leading "1"s. Both
choices can be defended. The patch at present does the second case, which is
similar what C++ does when casting signed to unsigned, returning e.g. 32 bits
or 64 bits but not something in between:

  int32_t c = -123456789;
  int64_t d = -123456789;
  std::cout << uint32_t (c) << ' ' << uint64_t (d) << '\n';


Please test against Matlab for the following cases:

dec2bin (-3)     # should return 8 bits
dec2bin (-4200)  # should return 16 bits
dec2bin (int16(-4200), 5)  # should hopefully still return 16 bits, not just
13 bits
dec2bin (intmin ("int64")) # should return one 1 followed by 63 zeroes.



(file #53747)

    _______________________________________________________

Additional Item Attachment:

File name: dec2bin.patch                  Size:2 KB
    <https://file.savannah.gnu.org/file/dec2bin.patch?file_id=53747>



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63089>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/