Re: [PATCH] parallel-tests: new recognized test result 'ERROR'

[Stefano Lattarini]

On Friday 01 July 2011, Stefano Lattarini wrote:
> While working on the next patch (which introduces a more rational testsuite
> summary report), I have made the `color*.test' tests even stricter (see
> attachements); and I think that it would be useful to have these stricter
> versions in this earlier change already.  I have thus amended the present
> patch accordingly.
> 
> Regards,
>   Stefano
> 
And then I've also squashed-in the attached diff, which fixes a minor bug
and enforces stricter checks with make implementations that aren't prone
to false positives.

I really hope this is the last tweak, really.

Thanks, and sorry for the noise.
   Stefano

diff --git a/tests/color.test b/tests/color.test
index 4ef85a4..7cbb080 100755
--- a/tests/color.test
+++ b/tests/color.test
@@ -98,16 +98,30 @@ test_color ()
 
 test_no_color ()
 {
-  {
-    # Not a useless use of cat; see above comments about grep.
-    cat stdout | grep "PASS.*:"
-    cat stdout | grep "FAIL:.*"
-    cat stdout | grep "SKIP.*:"
-    cat stdout | grep "XFAIL:.*"
-    cat stdout | grep "XPASS.*:"
-    cat stdout | grep "ERROR.*:"
-  } | grep "$esc" && Exit 1
-  :
+  # With make implementations that, like Solaris make, in case of errors
+  # print the whole failing recipe on standard output, we should content
+  # ourselves with a laxer check, to avoid false positives.
+  # Keep this in sync with lib/am/check.am:$(am__color_tests).
+  if $FGREP '= Xalways || test -t 1 ' stdout; then
+    # Extra verbose make, resort to laxer checks.
+    {
+      # Not a useless use of cat; see above comments about grep.
+      cat stdout | grep "PASS.*:"
+      cat stdout | grep "FAIL.*:"
+      cat stdout | grep "SKIP.*:"
+      cat stdout | grep "XFAIL.*:"
+      cat stdout | grep "XPASS.*:"
+      cat stdout | grep "ERROR.*:"
+      # To check that the testsuite summary is not unduly colorized.
+      cat stdout | grep '===='
+      cat stdout | grep 'test.*expected'
+      cat stdout | grep 'test.*not run'
+    } | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  else
+    cat stdout | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  fi
 }
 
 for vpath in false :; do
diff --git a/tests/color2.test b/tests/color2.test
index 74c993d..c283c53 100755
--- a/tests/color2.test
+++ b/tests/color2.test
@@ -111,16 +111,30 @@ test_color ()
 
 test_no_color ()
 {
-  {
-    # Not a useless use of cat; see above comments about grep.
-    cat stdout | grep "PASS.*:"
-    cat stdout | grep "FAIL:.*"
-    cat stdout | grep "SKIP.*:"
-    cat stdout | grep "XFAIL:.*"
-    cat stdout | grep "XPASS.*:"
-    cat stdout | grep "ERROR.*:"
-  } | grep "$esc" && Exit 1
-  :
+  # With make implementations that, like Solaris make, in case of errors
+  # print the whole failing recipe on standard output, we should content
+  # ourselves with a laxer check, to avoid false positives.
+  # Keep this in sync with lib/am/check.am:$(am__color_tests).
+  if $FGREP '= Xalways || test -t 1 ' stdout; then
+    # Extra verbose make, resort to laxer checks.
+    {
+      # Not a useless use of cat; see above comments about grep.
+      cat stdout | grep "PASS.*:"
+      cat stdout | grep "FAIL.*:"
+      cat stdout | grep "SKIP.*:"
+      cat stdout | grep "XFAIL.*:"
+      cat stdout | grep "XPASS.*:"
+      cat stdout | grep "ERROR.*:"
+      # To check that the testsuite summary is not unduly colorized.
+      cat stdout | grep '===='
+      cat stdout | grep 'test.*expected'
+      cat stdout | grep 'test.*not run'
+    } | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  else
+    cat stdout | grep "$esc" && Exit 1
+    : For shells with broken 'set -e'
+  fi
 }
 
 cat >expect-make <<'END'