[Qemu-devel] [PATCH] Fix FMULX not squashing denormalized inputs when FZ is set.

[Xiangyu Hu]

While FMULX returns a 2.0f float when two operators are infinity and
zero, those operators should be unpacked from raw inputs first. Inconsistent
cases would occur when operators are denormalized floats in flush-to-zero
mode. A wrong codepath will be entered and 2.0f will not be returned
without this patch.
Fix by checking whether inputs need to be flushed before running into
different codepaths.

Signed-off-by: Xiangyu Hu <address@hidden>
---
 target-arm/helper-a64.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/target-arm/helper-a64.c b/target-arm/helper-a64.c
index 81066ca..ebd9247 100644
--- a/target-arm/helper-a64.c
+++ b/target-arm/helper-a64.c
@@ -135,6 +135,9 @@ float32 HELPER(vfp_mulxs)(float32 a, float32 b, void *fpstp)
 {
     float_status *fpst = fpstp;
 
+    a = float32_squash_input_denormal(a, fpst);
+    b = float32_squash_input_denormal(b, fpst);
+
     if ((float32_is_zero(a) && float32_is_infinity(b)) ||
         (float32_is_infinity(a) && float32_is_zero(b))) {
         /* 2.0 with the sign bit set to sign(A) XOR sign(B) */
@@ -148,6 +151,9 @@ float64 HELPER(vfp_mulxd)(float64 a, float64 b, void *fpstp)
 {
     float_status *fpst = fpstp;
 
+    a = float64_squash_input_denormal(a, fpst);
+    b = float64_squash_input_denormal(b, fpst);
+
     if ((float64_is_zero(a) && float64_is_infinity(b)) ||
         (float64_is_infinity(a) && float64_is_zero(b))) {
         /* 2.0 with the sign bit set to sign(A) XOR sign(B) */
-- 
1.9.1