[Openexr-devel] [Offtopic]| half lossy datatype

[Gonzalo Garramuno]

I find myself with updating ffmpeg's exr reader (don't ask). One thing i 
am trying to do is add a gamma function to the reading of the format, 
with a gamma table.
However, I find myself against a wall not being able to get the float 
and half float to behave correctly.  I end up with greens in places 
where there were negative values or nans.
I am wondering if someone could take a look at these functions I am 
using to see if some bit is being set wrong.  Or if it is indeed 
impossible to have half->float->half (as done in my filling of the 
table) mantaining the same values (with gamma 1.0).

/**
 * Convert from 32-bit float as uint32_t to uint16_t.
 *
 * @param v 32-bit float
 *
 * @return normalized 16-bit unsigned int
 */
static inline uint16_t exr_flt2uint(uint32_t v)
{
    unsigned int exp = v >> 23;
    // "HACK": negative values result in exp<  0, so clipping them to 0
    // is also handled by this condition, avoids explicit check for 
sign bit.
    if (exp <= 127 + 7 - 24) // we would shift out all bits anyway
        return 0;
    if (exp >= 127)
        return 0xffff;
    v &= 0x007fffff;
    return (v + (1 << 23)) >> (127 + 7 - exp);
}


/**
 * Convert from 16-bit float as uint16_t to uint16_t.
 *
 * @param v 16-bit float
 *
 * @return normalized 16-bit unsigned int
 */
static inline uint16_t exr_halflt2uint(uint16_t v)
{
    unsigned exp = 14 - (v >> 10);
    if (exp >= 14) {
        if (exp == 14)
            return (v >> 9) & 1;
        else
            return (v & 0x8000) ? 0 : 0xffff;
    }
    v <<= 6;
    return (v + (1 << 16)) >> (exp + 1);
}

/*
 * Convert a half float as a uint16_t into a full float
 *
 */
static inline float exr_half2float(uint16_t h)
{
    static const union av_intfloat32 magic = { (254 - 15) << 23 };
    static const union av_intfloat32 was_infnan = { (127 + 16) << 23 };
    union  av_intfloat32 o;

    o.i = (h & 0x7fff) << 13; /* exponent/mantissa bits */
    o.f *= magic.f; /* exponent adjust */
    if (o.f >= was_infnan.f) /* make sure Inf/NaN survive */
        o.i |= 255 << 23;
    o.i |= (h & 0x8000) << 16; /* sign bit */

    return o.f;
}


For the gamma table, I am doing:

    float one_gamma = 1.0f / gamma;
    for ( uint32_t i = 0; i < 65536; ++i ) {
       t.f = exr_half2float(i);
       t.f = powf(t.f,one_gamma);
       s->gamma_table[i] = exr_flt2uint(t.i);
    }