[Octave-bug-tracker] [bug #58393] (buildbot) Tests are marked as "pass" if Octave crashes

[Markus Mützel]

URL:
  <https://savannah.gnu.org/bugs/?58393>

                 Summary: (buildbot) Tests are marked as "pass" if Octave
crashes
                 Project: GNU Octave
            Submitted by: mmuetzel
            Submitted on: Mon 18 May 2020 05:02:18 PM CEST
                Category: Configuration and Build System
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Incorrect Result
                  Status: None
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
                 Release: other
         Discussion Lock: Any
        Operating System: GNU/Linux

    _______________________________________________________

Details:

Like already reported on the mailing list a while ago:
https://octave.1599824.n4.nabble.com/buildbots-False-pass-results-for-segmentation-fault-in-test-td4695266.html

Quoting here:
> It looks like a segmentation fault in the last step of running the test
suite on the build bots is still marked as "success" if there were no failed
test up to that point. E.g.:
>
http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio
> 
> Maybe any output to stderr in that step should result in it to be marked as
"failure"? 

And a potential fix (but keep in mind that I don't know much about buildbot
configuration):

diff -r e02079ee69d1 master.cfg
--- a/master.cfg Fri Feb 08 15:58:06 2019 +0000
+++ b/master.cfg Mon Jan 13 19:44:13 2020 +0100
@@ -324,6 +324,7 @@
     self.passed = 0
     self.failed = 0
     self.warnings = 0
+    self.errors = 0
     self.summary_found = False
     self.complete = False
 
@@ -352,6 +353,11 @@
           self.rc = FAILURE
           self.failed += num
 
+  def errLineRecived (self, line)
+    self.total += 1
+    self.errors += 1
+    self.rc = FAILURE
+
 
 class octave_test (Test):
 
@@ -365,7 +371,7 @@
       passed = self.observer.total - self.observer.failed
 
       self.setTestResults (total = self.observer.total,
-                           failed = self.observer.failed,
+                           failed = self.observer.failed +
self.observer.errors,
                            passed = self.observer.passed,
                            warnings = self.observer.warnings) 





    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58393>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/