Re: [Qemu-devel] [PATCH] vnc: tight: Fix crash after 2GB of output

[Roland Dreier]

On Thu, Mar 3, 2011 at 11:34 PM, Michael Tokarev <address@hidden> wrote:
>> The fix for this is simple: keep previous_out as a uLong too, which
>> avoids any problems with sign conversion or truncation.
>
> This looks wrong to me.  On 32bit x86 uLong is 32bits.  Yes
> it's unsigned there, but it's still 32bits.  And sure thing
> it _will_ overflow again, not at 2Gb but now at 4Gb, and the
> next total_out will start from zero again, so that now bytes
> will be a large integer again because "next" (new total_out)
> will be less than "current" (previous_out).

Actually there is no problem with overflow of unsigned long.
The C standard says that unsigned arithmetic is simply done
modulo the size of the integer, so when total_out reaches
4GB, things will just wrap around (and the difference
between "nearby" values will still be the correct, small
value).  For example, if previous were (4GB - 5) and
then total_out had 1000 added to it, total_out would
end up as 995, and total_out - previous would be 1000.

In other words I'm pretty sure my fix works on 32 bits too.

However your fix using avail_out is probably better anyway,
and your latest patch fixes both instances of the bug, so
we might as well go with that.

Thanks,
 - R.