Re: [Qemu-devel] [PATCH] vnc: Fix tight_detect_smooth_image() for lossless case

[Markus Armbruster]

Ping?

Markus Armbruster <address@hidden> writes:

> VncTight member uint8_t quality is either (uint8_t)-1 for lossless or
> less than 10 for lossy.
>
> tight_detect_smooth_image() first promotes it to int, then compares
> with -1.  Always unequal, so we always execute the lossy code.  Reads
> beyond tight_conf[] and returns crap when quality is actually
> lossless.
>
> Compare to (uint8_t)-1 instead, like we do elsewhere.
>
> Spotted by Coverity.
>
> Signed-off-by: Markus Armbruster <address@hidden>
> ---
>  ui/vnc-enc-tight.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c
> index e6966ae..59b59c0 100644
> --- a/ui/vnc-enc-tight.c
> +++ b/ui/vnc-enc-tight.c
> @@ -330,7 +330,7 @@ tight_detect_smooth_image(VncState *vs, int w, int h)
>      } else {
>          errors = tight_detect_smooth_image16(vs, w, h);
>      }
> -    if (quality != -1) {
> +    if (quality != (uint8_t)-1) {
>          return (errors < tight_conf[quality].jpeg_threshold);
>      }
>      return (errors < tight_conf[compression].gradient_threshold);