Re: Strange tests failure on Travis-CI due to AM_TESTS_ENVIRONMENT

[Sami Kerola]

On 12 March 2014 04:25, Assaf Gordon <address@hidden> wrote:
> I'm trying to convert my project non-recursive makefile, and encountered a
> strange problem with the tests.
>
> I managed to reproduce and isolate it using GNU hello, so I figured perhaps
> you can help me.
>
> The problem is this:
> 1. GNU Hello, cloned from latest repository.
> 2. locally (on old ubuntu), "make check" works just fine.
> 3. Using "Travis-CI" (the website for continuous integration), the tests
> fail.
>
> I've managed to pinpoint the issue to the following:
> Inside Makefile.am, the test environment is set using AM_TESTS_ENVIRONMENT.
> For some strange reason, this is ignored/bypassed on Travis-CI's servers
> (They are using Ubuntu 12.04 which should be standard).
> When changed to TESTS_ENVIRONMENT, the tests pass.
>
> =========
> The steps I've used:
> 1. Cloned GNU Hello, put in GitHub, setup Travis-CI.
> "make check" fails, all tests can't find the "hello" executable.
>
> This is the log, scroll to line 2139 to see the tests results:
>   https://travis-ci.org/agordon/gnuhello/jobs/20581026#L2139
>
> 2. Modified "tests/greeting-2", to print the environment and always fail:
> https://github.com/agordon/gnuhello/commit/6a5af85e99afe69cdec6ccb015ccd9b95502bff4
>
> This is the log, scroll to line 2168 to see the printed environment:
>   https://travis-ci.org/agordon/gnuhello/jobs/20581280#L2168
> Note especially, the PATH does not contain "." and there is no "HELLO"
> variable (both are set in AM_TESTS_ENVIRONMENT).
>
> 3. Modified "Makefile.am", changed "AM_TESTS_ENVIRONMENT" to
> "TESTS_ENVIRONMENT"
> https://github.com/agordon/gnuhello/commit/0b54a2fee303f42f85f99284e00b8aad3a667232
>
> This is the log after the change, scroll to line 2161 to see the updated
> printed environment:
> https://travis-ci.org/agordon/gnuhello/jobs/20581441#L2161
>
> After the change, PATH is valid and HELLO is set, and the tests pass (except
> 'greeting-2' which fails on purpose).
> =========
>
> So my question -
> What's going on, and is changing "AM_TESTS_ENVIRONMENT" the correct
> solution?

Hi Gordon,

Thank you for good bug report.

Short answer: changing AM_TESTS_ENVIRONMENT seems to be the correct
solution.

Long answer: unless someone objects I'll push the following to upstream.

--->8----
From: Sami Kerola <address@hidden>
Date: Thu, 13 Mar 2014 17:18:36 +0000
Subject: [PATCH] build: parallel-tests require of TESTS_ENVIRONMENT

* Makefile.am: Tests failed in Travis-CI's servers because missing
environment variables.  Looking the automake documentation the
AM_TESTS_ENVIRONMENT cannot be used in combination with parallel-tests,
and such cases TESTS_ENVIRONMENT should be used instead.

Reported by Assaf Gordon.
https://www.gnu.org/software/automake/manual/html_node/Serial-Test-Harness.html
---
 Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index eb02a4c..6285887 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -68,7 +68,7 @@ BUILT_SOURCES =
 
 LOG_COMPILER = $(SHELL)
 
-AM_TESTS_ENVIRONMENT = \
+TESTS_ENVIRONMENT = \
   top_srcdir=$(top_srcdir) \
   PATH=.$(PATH_SEPARATOR)$(PATH_SEPARATOR)$$PATH \
   HELLO=`echo hello | sed '$(transform)'`; \
-- 
1.9.0