[Qemu-devel] Hang bug in 80-bit float square root implementation

[Andrew Dutcher]

Hello!

I ran into an issue where qemu (specifically, the unicorn engine)
would hang forever in the middle of the emulated square root
instruction under certain circumstances. I eventually tracked the
issue down to the square root of an "unnormal" long double, one
without the integer part bit set.

Test case:

cat > hang.s <<EOF
.intel_syntax noprefix

.global _start
_start:
  xor eax,eax
  inc eax
  push eax
  push eax
  push eax
  fld tbyte ptr [esp]
  fsqrt
  int 0x80
EOF
as --32 hang.s -o hang.o
ld -melf_i386 hang.o -o hang
qemu-i386 ./hang

qemu will hang! Assuming it jits the fsqrt as the soft float
implementation and doesn't just use a native fsqrt instruction. I have
no idea if qemu can do this, but it doesn't hurt to cover that base.

Here is my patch to the bug. It attempts to normalize the unnormal
floats before square rooting them. I don't know if it makes it produce
the correct results, but I also don't know if there's any good ground
truth for what the result should be, since my understanding is that
different x86 family processors handle unnormal floats differently.

I've never submitted a patch here before, so I hope this is the right
way to do it!

diff --git a/fpu/softfloat.c b/fpu/softfloat.c
index 9b1eccf..75a757d 100644
--- a/fpu/softfloat.c
+++ b/fpu/softfloat.c
@@ -5572,8 +5572,12 @@ floatx80 floatx80_sqrt(floatx80 a, float_status *status)
         float_raise(float_flag_invalid, status);
         return floatx80_default_nan(status);
     }
+    if ( aSig0 == 0 ) return packFloatx80( 0, 0, 0 );
+    while (aExp != 0 && (aSig0 & 0x8000000000000000) == 0) {
+        aExp--;
+        aSig0 <<= 1;
+    }
     if ( aExp == 0 ) {
-        if ( aSig0 == 0 ) return packFloatx80( 0, 0, 0 );
         normalizeFloatx80Subnormal( aSig0, &aExp, &aSig0 );
     }
     zExp = ( ( aExp - 0x3FFF )>>1 ) + 0x3FFF;