[bug #15942] test_driver.pl - run_each_test() fails to set $status when test dies, prints value of $status from prior test

[greg keranen]

URL:
  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15942>

                 Summary: test_driver.pl - run_each_test() fails to set
$status when test dies, prints value of $status from prior test
                 Project: make
            Submitted by: gkeranen
            Submitted on: Wed 03/01/06 at 11:26
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
       Component Version: CVS
        Platform Version: Any
           Fixed Release: None

    _______________________________________________________

Details:

--- TEST ENVIRONMENT:
GNU Make 3.81rc1 on CYGWIN_NT-5.1 x2 1.5.19(0.150/4/2) i686

--- STEPS TO REPRODUCE
cd make-3.81rc1
make check

    Finding tests...
    [...]
    features/default_names ..................................
    *** Test died (features/default_names): test_driver.pl: 453: abort at
test_drive
    r.pl line 638.
    
    ok     (4 passed)

--- EXPECTED OUTPUT:
status should be set to "N/A"
    
--- ANALYSIS:
toplevel reports success 'ok (4 passed)' when default_names dies; should not
say 'ok'. 
On success, if $port_type eq 'UNIX', default_names should print:  'ok (3
passed)'
- this is the maximum number of tests that this script runs
On success, if $port_type eq 'W32', default_names should print:  'ok (2
passed)'

$status is not set at the top of the run_each_test() and neither is it set in
the case where !defined($code)

--- CODE CAUSING THE PROBLEM:
tests/test_driver.pl function run_each_test()

[...]
    # How did it go?
    if (!defined($code))
    {
      $suite_passed = 0;

[...]
    }
    
    if ($suite_passed) {
      ++$categories_passed;
      $status = "ok     ($tests_passed passed)";
[...]
    }
    elsif ($code > 0) {
      $status = "FAILED ($tests_passed/$tests_run passed)";
    }
    elsif ($code < 0) {
      $status = "N/A";
      --$categories_run;
    }


--- PROPOSED FIX:
File attached: tests/test_driver.pl with modified function run_each_test()

NOTE:
    OLD:
        $status = "N/A";
    NEW:
        $status = "status = N/A";
    REASON:
        $status is only used for direct display to the user; simply printing
"N/A" on a line by itself does not convey any clear meaning.

    # Run the actual test!
    $tests_run = 0;
    $tests_passed = 0;
    # 2006.02.28 address@hidden added following line:
    $status = "status = N/A";
    $code = do $perl_testname;
[...]

    # 2006.02.28 address@hidden
    # OLD: elsif ($code < 0) {
    elsif (($code < 0) || !defined($code) ) {
      $status = "status = N/A";
      --$categories_run;
    }

--- REGRESSION TEST 
Proposed fix was applied and default_name set to die by run_make_tests.pl:

FILE: run_make_tests.pl set to force default_names to die:
   if ($osname =~ /Windows/i || $osname =~ /MINGW32/i ) {

OUTPUT from make check:
[...]
features/default_names .................................. Error running
/cygdriv
e/c/greg/src/make-3.81rc1/tests/../make (expected 0; got 512):
/cygdrive/c/greg/
src/make-3.81rc1/tests/../make  --no-print-directory -C
/cygdrive/c/greg/src/mak
e-3.81rc1/tests/work/

*** Test died (features/default_names): test_driver.pl: 455: compare_output
fail
ed: Makefile at test_driver.pl line 643.

status = N/A
[...]






    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Wed 03/01/06 at 11:26  Name: test_driver.pl  Size: 29.54KB   By:
gkeranen
tests/test_driver.pl with modified function run_each_test()
<http://savannah.gnu.org/bugs/download.php?item_id=15942&item_file_id=3458>

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?func=detailitem&item_id=15942>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/