Re: [buildbots] False pass results for segmentation fault in test

[Markus Mützel]

Am 13. Januar 2020 um 19:46 Uhr schrieb "Markus Mützel":
> Am 13. Januar 2020 um 16:21 Uhr schrieb "Markus Mützel":
> > It looks like a segmentation fault in the last step of running the test 
> > suite on the build bots is still marked as "success" if there were no 
> > failed test up to that point. E.g.:
> > http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio
> > 
> > Maybe any output to stderr in that step should result in it to be marked as 
> > "failure"?
> 
> I only have a very vague idea of how configuring the buildbots works. And I 
> have no way of testing this. But maybe something similar to this change would 
> work:
> 
> diff -r e02079ee69d1 master.cfg
> --- a/master.cfg      Fri Feb 08 15:58:06 2019 +0000
> +++ b/master.cfg      Mon Jan 13 19:44:13 2020 +0100
> @@ -324,6 +324,7 @@
>      self.passed = 0
>      self.failed = 0
>      self.warnings = 0
> +    self.errors = 0
>      self.summary_found = False
>      self.complete = False
>  
> @@ -352,6 +353,11 @@
>            self.rc = FAILURE
>            self.failed += num
>  
> +  def errLineReceived (self, line)
> +    self.total += 1
> +    self.errors += 1
> +    self.rc = FAILURE
> +
>  
>  class octave_test (Test):
>  
> @@ -365,7 +371,7 @@
>        passed = self.observer.total - self.observer.failed
>  
>        self.setTestResults (total = self.observer.total,
> -                           failed = self.observer.failed,
> +                           failed = self.observer.failed + 
> self.observer.errors,
>                             passed = self.observer.passed,
>                             warnings = self.observer.warnings)


These errors still occur occasionally. E.g. here: 
http://buildbot.octave.org:8010/#/builders/12/builds/1599/steps/6/logs/stdio

Since the overall test appears as "green", they are quite hard to spot.

Could be please apply a change similar to the one outlined above to the 
buildbot master?

Should I better open a bug report?

Markus