From MAILER-DAEMON Mon May 05 14:24:37 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1Jt5Mb-0007Wm-6p for mharc-autoconf-patches@gnu.org; Mon, 05 May 2008 14:24:37 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Jt5MZ-0007WP-Ed for autoconf-patches@gnu.org; Mon, 05 May 2008 14:24:35 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Jt5MX-0007Vg-PL for autoconf-patches@gnu.org; Mon, 05 May 2008 14:24:34 -0400 Received: from [199.232.76.173] (port=51807 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Jt5MX-0007VY-C5 for autoconf-patches@gnu.org; Mon, 05 May 2008 14:24:33 -0400 Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Jt5MW-0003RI-7e for autoconf-patches@gnu.org; Mon, 05 May 2008 14:24:32 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Jt5MS-0003St-Rp for autoconf-patches@gnu.org; Mon, 05 May 2008 18:24:28 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 05 May 2008 18:24:28 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 05 May 2008 18:24:28 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Mon, 5 May 2008 18:24:20 +0000 (UTC) Lines: 204 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: testsuite 01 X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 May 2008 18:24:35 -0000 I got bit when trying to rerun just failed tests: $ ./testsuite `cd ./testsuite.dir; echo [0-9]*` ## --------------------------------------- ## ## GNU M4 1.9a.413-359f9-dirty test suite. ## ## --------------------------------------- ## ./testsuite: error: cannot find myself; rerun with an absolute file name I traced it to the fact that './testsuite 01' causes at_func_test to try to use the undefined $at_sed01 rather than $at_sed1. Any objections to this patch, which normalizes away leading 0 up front? Also, this points out something to be careful of - at_func_arith treats leading 0 differently (octal vs. decimal) depending on whether the shell supports $(()). >From cf161a191e197d8cb494ef6834e1393ce81ebd38 Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Mon, 5 May 2008 12:16:07 -0600 Subject: [PATCH] Improve behavior of 'tests/testsuite 01'. * lib/autotest/general.m4 (AT_INIT) : Alter usage to eval its argument, in order to normalize away leading zero. All callers updated. * tests/autotest.at (Keywords and ranges): Test range normalization with leading 0. Signed-off-by: Eric Blake --- ChangeLog | 9 +++++++++ lib/autotest/general.m4 | 28 +++++++++++++++++++--------- tests/autotest.at | 36 +++++++++++++++++++++++++++++------- 3 files changed, 57 insertions(+), 16 deletions(-) diff --git a/ChangeLog b/ChangeLog index 705e71e..5add251 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,12 @@ +2008-05-05 Eric Blake + + Improve behavior of 'tests/testsuite 01'. + * lib/autotest/general.m4 (AT_INIT) : + Alter usage to eval its argument, in order to normalize away + leading zero. All callers updated. + * tests/autotest.at (Keywords and ranges): Test range + normalization with leading 0. + 2008-04-26 Eric Blake Mention Solaris /usr/ucb/tr pitfall. diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4 index 4ce3df7..1b1605b 100644 --- a/lib/autotest/general.m4 +++ b/lib/autotest/general.m4 @@ -440,17 +440,27 @@ at_format='m4_bpatsubst(m4_defn([AT_ordinal]), # Description of all the test groups. at_help_all="AS_ESCAPE(m4_dquote(m4_defn([AT_help_all])))" -# at_func_validate_ranges [N...] -# ------------------------------ -# validate test group ranges +# at_func_validate_ranges [NAME...] +# --------------------------------- +# Validate and normalize the test group number contained in each NAME. at_func_validate_ranges () { for at_grp do - if test $at_grp -lt 1 || test $at_grp -gt AT_ordinal; then - AS_ECHO(["invalid test group: $at_grp"]) >&2 + eval at_value=\$$at_grp + if test $at_value -lt 1 || test $at_value -gt AT_ordinal; then + AS_ECHO(["invalid test group: $at_value"]) >&2 exit 1 fi + case $at_value in + 0*) # We want to treat leading 0 as decimal, like expr and test, but + # at_func_arith treats it as octal if it uses $(( )). + # With XSI shells, we could avoid the expr fork by using + # ${at_value#${at_value%%[1-9]*}}, but it is not worth the effort + # when the user can just avoid requesting tests with leading 0. + eval $at_grp='`expr $at_value + 0`' + ;; + esac done }])])dnl m4_divert_push([PARSE_ARGS])dnl @@ -506,14 +516,14 @@ do ;; [[0-9] | [0-9][0-9] | [0-9][0-9][0-9] | [0-9][0-9][0-9][0-9]]) - at_func_validate_ranges $at_option + at_func_validate_ranges at_option at_groups="$at_groups$at_option " ;; # Ranges [[0-9]- | [0-9][0-9]- | [0-9][0-9][0-9]- | [0-9][0-9][0-9][0-9]-]) at_range_start=`echo $at_option |tr -d X-` - at_func_validate_ranges $at_range_start + at_func_validate_ranges at_range_start at_range=`AS_ECHO([" $at_groups_all "]) | \ sed -e 's/^.* \('$at_range_start' \)/\1/'` at_groups="$at_groups$at_range " @@ -521,7 +531,7 @@ do [-[0-9] | -[0-9][0-9] | -[0-9][0-9][0-9] | -[0-9][0-9][0-9][0-9]]) at_range_end=`echo $at_option |tr -d X-` - at_func_validate_ranges $at_range_end + at_func_validate_ranges at_range_end at_range=`AS_ECHO([" $at_groups_all "]) | \ sed -e 's/\( '$at_range_end'\) .*$/\1/'` at_groups="$at_groups$at_range " @@ -540,7 +550,7 @@ do at_range_end=$at_range_start at_range_start=$at_tmp fi - at_func_validate_ranges $at_range_start $at_range_end + at_func_validate_ranges at_range_start at_range_end at_range=`AS_ECHO([" $at_groups_all "]) | \ sed -e 's/^.*\( '$at_range_start' \)/\1/' \ -e 's/\( '$at_range_end'\) .*$/\1/'` diff --git a/tests/autotest.at b/tests/autotest.at index 7db8451..5571eef 100644 --- a/tests/autotest.at +++ b/tests/autotest.at @@ -677,22 +677,41 @@ AT_SETUP([Keywords and ranges]) AT_CHECK_AT_PREP([k], [[AT_INIT -AT_SETUP(none) +AT_SETUP(none) #01 AT_CHECK(:) AT_CLEANUP -AT_SETUP(first) +AT_SETUP(first) #02 AT_KEYWORDS(key1) AT_CHECK(:) AT_CLEANUP -AT_SETUP(second) +AT_SETUP(second) #03 AT_KEYWORDS(key2) AT_CHECK(:) AT_CLEANUP -AT_SETUP(both) +AT_SETUP(both) #04 AT_KEYWORDS([key1 key2]) AT_KEYWORDS([key1]) AT_CHECK(:) AT_CLEANUP +AT_SETUP(test5) #05 +AT_CHECK(:) +AT_CLEANUP +AT_SETUP(test6) #06 +AT_CHECK(:) +AT_CLEANUP +AT_SETUP(test7) #07 +AT_CHECK(:) +AT_CLEANUP +AT_SETUP(test8) #08 +AT_CHECK(:) +AT_CLEANUP +AT_SETUP(test9) #09 +AT_CHECK(:) +AT_CLEANUP +AT_SETUP(test10) #10 +AT_KEYWORDS([final]) +AT_CHECK(:) +AT_CLEANUP ]]) dnl check that AT_KEYWORDS does not duplicate words AT_CHECK([grep 'key1.*key1' k], [1]) @@ -733,21 +752,24 @@ AT_CHECK_KEYS([-k 'eco.*'], [successful], [1], AT_CHECK_KEYS([-k 'fir.*'], [first], [1], [none|second|both], [0]) AT_CHECK_KEYS([1-2], [none|first], [2], [second|both], [0]) +AT_CHECK_KEYS([01-002], [none|first], [2], [second|both], [0]) AT_CHECK_KEYS([1-3 2-1], [none|first|second], [3], [both], [0]) AT_CHECK_KEYS([-3], [none|first|second], [3], [both], [0]) AT_CHECK_KEYS([4-], [both], [1], [none|first|second], [0]) +AT_CHECK_KEYS([010], [test10], [1], [none|first|second|both], [0]) AT_CHECK_KEYS([-k second 4-], [second|both], [2], [none|first], [0]) AT_CHECK([$CONFIG_SHELL ./k 0], [1], [ignore], [ignore]) AT_CHECK([$CONFIG_SHELL ./k 0-], [1], [ignore], [ignore]) AT_CHECK([$CONFIG_SHELL ./k -0], [1], [ignore], [ignore]) -AT_CHECK([$CONFIG_SHELL ./k 5], [1], [ignore], [ignore]) -AT_CHECK([$CONFIG_SHELL ./k 5-], [1], [ignore], [ignore]) -AT_CHECK([$CONFIG_SHELL ./k 1-5], [1], [ignore], [ignore]) +AT_CHECK([$CONFIG_SHELL ./k 11], [1], [ignore], [ignore]) +AT_CHECK([$CONFIG_SHELL ./k 11-], [1], [ignore], [ignore]) +AT_CHECK([$CONFIG_SHELL ./k 1-11], [1], [ignore], [ignore]) AT_CHECK([$CONFIG_SHELL ./k -k nonexistent], [0], [ignore]) AT_CHECK_KEYS([--list -k nonexistent], [KEYWORDS], [1], [first|second|both], [0]) AT_CHECK_KEYS([--list 1], [none], [1], [first|second|both], [0]) +AT_CHECK_KEYS([--list 01], [none], [1], [first|second|both], [0]) AT_CHECK_KEYS([--list -k none -k first], [none|first], [2], [second|both], [0]) AT_CLEANUP -- 1.5.5.1 From MAILER-DAEMON Mon May 05 23:11:44 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JtDai-0005hM-9t for mharc-autoconf-patches@gnu.org; Mon, 05 May 2008 23:11:44 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JtDag-0005fb-5U for autoconf-patches@gnu.org; Mon, 05 May 2008 23:11:42 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JtDae-0005db-Ee for autoconf-patches@gnu.org; Mon, 05 May 2008 23:11:41 -0400 Received: from [199.232.76.173] (port=56988 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JtDae-0005dP-1S for autoconf-patches@gnu.org; Mon, 05 May 2008 23:11:40 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JtDae-0003xM-Bg for autoconf-patches@gnu.org; Mon, 05 May 2008 23:11:40 -0400 Received: from localhost.localdomain (xdsl-87-78-70-237.netcologne.de [87.78.70.237]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id DC852400014F3; Tue, 6 May 2008 05:11:38 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1JtDaT-0003jZ-8y; Tue, 06 May 2008 05:11:29 +0200 Date: Tue, 6 May 2008 05:11:29 +0200 From: Ralf Wildenhues To: Eric Blake Message-ID: <20080506031128.GA24346@ins.uni-bonn.de> Mail-Followup-To: Eric Blake , autoconf-patches@gnu.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Cc: autoconf-patches@gnu.org Subject: Re: testsuite 01 X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2008 03:11:42 -0000 Hi Eric, * Eric Blake wrote on Mon, May 05, 2008 at 08:24:20PM CEST: > Subject: [PATCH] Improve behavior of 'tests/testsuite 01'. > > * lib/autotest/general.m4 (AT_INIT) : > Alter usage to eval its argument, in order to normalize away > leading zero. All callers updated. > * tests/autotest.at (Keywords and ranges): Test range > normalization with leading 0. OK, thanks! > +# at_func_validate_ranges [NAME...] > +# --------------------------------- > +# Validate and normalize the test group number contained in each NAME. Maybe: "in each variable NAME"? Otherwise, I had difficulties understanding this. > at_func_validate_ranges () > { Cheers, Ralf From MAILER-DAEMON Tue May 06 08:21:23 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JtMAd-0001bI-2C for mharc-autoconf-patches@gnu.org; Tue, 06 May 2008 08:21:23 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JtMAb-0001bD-M1 for autoconf-patches@gnu.org; Tue, 06 May 2008 08:21:21 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JtMAb-0001b1-6e for autoconf-patches@gnu.org; Tue, 06 May 2008 08:21:21 -0400 Received: from [199.232.76.173] (port=49113 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JtMAb-0001ay-4M for autoconf-patches@gnu.org; Tue, 06 May 2008 08:21:21 -0400 Received: from qmta01.emeryville.ca.mail.comcast.net ([76.96.30.16]) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JtMAa-0000kr-L4 for autoconf-patches@gnu.org; Tue, 06 May 2008 08:21:20 -0400 Received: from OMTA02.emeryville.ca.mail.comcast.net ([76.96.30.19]) by QMTA01.emeryville.ca.mail.comcast.net with comcast id N9z41Z0020QkzPwA10C800; Tue, 06 May 2008 12:21:12 +0000 Received: from [192.168.0.103] ([67.166.125.73]) by OMTA02.emeryville.ca.mail.comcast.net with comcast id NCMA1Z0021b8C2B8N00000; Tue, 06 May 2008 12:21:12 +0000 X-Authority-Analysis: v=1.0 c=1 a=xe8BsctaAAAA:8 a=sY3RoIn2ok3C1QQlaIYA:9 a=J9bGVvoxOjwzd5eALIoA:7 a=gdZ_F_-bDRs-kvg2xddfntX2sT0A:4 a=eDFNAWYWrCwA:10 a=rPt6xJ-oxjAA:10 Message-ID: <48204D3D.7040606@byu.net> Date: Tue, 06 May 2008 06:21:17 -0600 From: Eric Blake User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080421 Thunderbird/2.0.0.14 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: autoconf-patches@gnu.org References: <20080506031128.GA24346@ins.uni-bonn.de> In-Reply-To: <20080506031128.GA24346@ins.uni-bonn.de> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized. Subject: Re: testsuite 01 X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2008 12:21:21 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 According to Ralf Wildenhues on 5/5/2008 9:11 PM: |> Subject: [PATCH] Improve behavior of 'tests/testsuite 01'. |> | | OK, thanks! | |> +# at_func_validate_ranges [NAME...] |> +# --------------------------------- |> +# Validate and normalize the test group number contained in each NAME. | | Maybe: "in each variable NAME"? Otherwise, I had difficulties | understanding this. Yes, adding the word 'variable' made more sense. I committed the updated patch. - -- Don't work too hard, make some time for fun as well! Eric Blake ebb9@byu.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkggTTwACgkQ84KuGfSFAYCD3gCePGBEiSBf5DmbJLPv/Z0qs5ts 1JMAn1iQuwE5zH4HMieGo/VqLE8hvBrP =4u29 -----END PGP SIGNATURE----- From MAILER-DAEMON Tue May 06 11:53:50 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JtPUE-0008Q6-OD for mharc-autoconf-patches@gnu.org; Tue, 06 May 2008 11:53:50 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JtPUC-0008PA-C8 for autoconf-patches@gnu.org; Tue, 06 May 2008 11:53:48 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JtPU8-0008NT-Eb for autoconf-patches@gnu.org; Tue, 06 May 2008 11:53:47 -0400 Received: from [199.232.76.173] (port=52227 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JtPU7-0008NN-Ub for autoconf-patches@gnu.org; Tue, 06 May 2008 11:53:43 -0400 Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1JtPU6-0003Mn-Pb for autoconf-patches@gnu.org; Tue, 06 May 2008 11:53:43 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JtPU0-0005dm-VU for autoconf-patches@gnu.org; Tue, 06 May 2008 15:53:36 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 06 May 2008 15:53:36 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 06 May 2008 15:53:36 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Tue, 6 May 2008 15:53:25 +0000 (UTC) Lines: 96 Message-ID: References: <20080506031128.GA24346@ins.uni-bonn.de> <48204D3D.7040606@byu.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Re: testsuite 01 X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2008 15:53:49 -0000 Eric Blake byu.net> writes: > |> Subject: [PATCH] Improve behavior of 'tests/testsuite 01'. > |> > > Yes, adding the word 'variable' made more sense. I committed the updated > patch. > I'm also committing this, after playing more with $(()). It looks like pdksh, at least version 5.2.14, has a bug even when using 'set -o posix'. And zsh behaves differently depending on whether it is in posix mode. Bash and posh both met expectations without any extra effort. >From 2811da8583226d60b8082852f8f6085e8c47bd3d Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Tue, 6 May 2008 09:31:55 -0600 Subject: [PATCH] Document $(( )) pitfalls. * doc/autoconf.texi (Shell Substitutions): Mention octal vs. decimal. Mention autotest's at_func_arith. Signed-off-by: Eric Blake --- ChangeLog | 4 ++++ doc/autoconf.texi | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 0 deletions(-) diff --git a/ChangeLog b/ChangeLog index 59a3119..c2d0f17 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,9 @@ 2008-05-06 Eric Blake + Document $(( )) pitfalls. + * doc/autoconf.texi (Shell Substitutions): Mention octal + vs. decimal. Mention autotest's at_func_arith. + Improve behavior of './testsuite 01'. * lib/autotest/general.m4 (AT_INIT) : Alter usage to eval its arguments, in order to normalize away diff --git a/doc/autoconf.texi b/doc/autoconf.texi index a5e880a..cd9a389 100644 --- a/doc/autoconf.texi +++ b/doc/autoconf.texi @@ -13203,6 +13203,44 @@ echo $(case x in x) echo hello;; esac) Arithmetic expansion is not portable as some shells (most notably Solaris 10 @command{/bin/sh}) don't support it. +Among shells that do support @samp{$(( ))}, not all of them obey the +Posix rule that octal and hexadecimal constants must be recognized: + +@example +$ @kbd{bash -c 'echo $(( 010 + 0x10 ))'} +24 +$ @kbd{zsh -c 'echo $(( 010 + 0x10 ))'} +26 +$ @kbd{zsh -c 'emulate sh; echo $(( 010 + 0x10 ))'} +24 +$ @kbd{pdksh -c 'echo $(( 010 + 0x10 ))'} +pdksh: 010 + 0x10 : bad number `0x10' +$ @kbd{pdksh -c 'echo $(( 010 ))'} +10 +@end example + +When it is available, using arithmetic expansion provides a noticeable +speedup in script execution; but testing for support requires +@command{eval} to avoid syntax errors. If shell function support has +also been detected, then this construct can be used to assign @samp{foo} +to an arithmetic result, provided all numeric arguments are provided in +decimal and without a leading zero: + +@example +if ( eval 'test $(( 1 + 1 )) = 2' ) 2>/dev/null; then + eval 'func_arith () + @{ + func_arith_result=$(( $* )) + @}' +else + at_func_arith () + @{ + func_arith_result=`expr "$@@"` + @} +fi +func_arith 1 + 1 +foo=$func_arith_result +@end example @item ^ -- 1.5.5.1 From MAILER-DAEMON Tue May 06 16:42:11 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JtTzH-0005Fs-0x for mharc-autoconf-patches@gnu.org; Tue, 06 May 2008 16:42:11 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JtTzG-0005Fm-6W for autoconf-patches@gnu.org; Tue, 06 May 2008 16:42:10 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JtTzB-0005FC-CA for autoconf-patches@gnu.org; Tue, 06 May 2008 16:42:09 -0400 Received: from [199.232.76.173] (port=50020 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JtTzB-0005F7-8s for autoconf-patches@gnu.org; Tue, 06 May 2008 16:42:05 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JtTzA-0006n8-MK for autoconf-patches@gnu.org; Tue, 06 May 2008 16:42:04 -0400 Received: from localhost.localdomain (xdsl-87-78-109-22.netcologne.de [87.78.109.22]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id A3F694000073F for ; Tue, 6 May 2008 22:42:03 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1JtTz0-0004Hp-7X for autoconf-patches@gnu.org; Tue, 06 May 2008 22:41:54 +0200 Date: Tue, 6 May 2008 22:41:54 +0200 From: Ralf Wildenhues To: autoconf-patches@gnu.org Message-ID: <20080506204153.GC16385@ins.uni-bonn.de> Mail-Followup-To: autoconf-patches@gnu.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Subject: let AC_MSG_FAILURE report pwd X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2008 20:42:10 -0000 This is probably the most often misunderstood build failure of GCC: http://gcc.gnu.org/ml/gcc-help/2008-05/msg00058.html http://article.gmane.org/gmane.comp.gcc.help/23852 http://thread.gmane.org/gmane.comp.gcc.help/24232 to list just the reports from the last few weeks. People simply don't realize that there could be more than one config.log and config.cache file. OK to apply? I considered propagating relative subdirs to AC_CONFIG_SUBDIRS, but that doesn't seem worth the trouble (and won't work OOTB with GCC). Thanks, Ralf * lib/autoconf/general.m4 (_AC_ARG_VAR_VALIDATE, AC_MSG_FAILURE): Output $ac_pwd along with fatal failure. * tests/torture.at (Deep Package): Extend test. Reported numerous times against GCC, and probably other packages. diff --git a/lib/autoconf/general.m4 b/lib/autoconf/general.m4 index 57b4871..0f8e32d 100644 --- a/lib/autoconf/general.m4 +++ b/lib/autoconf/general.m4 @@ -1576,6 +1576,7 @@ for ac_var in $ac_precious_vars; do fi done if $ac_cache_corrupted; then + AS_MESSAGE([error: in `$ac_pwd':], 2) AS_MESSAGE([error: changes in the environment can compromise the build], 2) AS_ERROR([run `make distclean' and/or `rm $cache_file' and start over]) fi])dnl @@ -2166,8 +2167,9 @@ m4_copy([AS_WARN], [AC_MSG_WARN]) m4_copy([AS_MESSAGE], [AC_MSG_NOTICE]) m4_copy([AS_ERROR], [AC_MSG_ERROR]) m4_define([AC_MSG_FAILURE], -[AC_MSG_ERROR([$1 -See `config.log' for more details.], [$2])]) +[{ AS_MESSAGE([error: in `$ac_pwd':], 2) +AC_MSG_ERROR([$1 +See `config.log' for more details.], [$2]); }]) # _AC_MSG_LOG_CONFTEST diff --git a/tests/torture.at b/tests/torture.at index 4b7240b..d5ec2e8 100644 --- a/tests/torture.at +++ b/tests/torture.at @@ -1275,6 +1275,9 @@ AT_DATA([inner/innermost/configure.in], AC_ARG_VAR([INNERMOST], [an innermost variable]) AC_CONFIG_HEADERS(config.h:config.hin) AC_DEFINE_UNQUOTED([INNERMOST], [$INNERMOST], [an innermost variable]) +if test -n "$innermost_error"; then + AC_MSG_FAILURE([Error in $PACKAGE_NAME]) +fi AC_OUTPUT ]]) @@ -1324,6 +1327,14 @@ AT_CHECK([grep INNERMOST inner/innermost/config.h], 0, [[#define INNERMOST tsomrenni ]]) +# Ensure we point to the right config.log file for errors. +AT_CHECK_CONFIGURE([innermost_error=:], [1], [], [stderr]) +AT_CHECK([grep 'inner/innermost' stderr], [], [ignore]) +cd inner +AT_CHECK_CONFIGURE([innermost_error=:], [1], [], [stderr]) +AT_CHECK([grep 'innermost' stderr], [], [ignore]) +cd .. + # The same, but from a builddir. AS_MKDIR_P([builddir]) chmod a-w builddir inner/innermost inner From MAILER-DAEMON Tue May 06 18:01:18 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JtVDp-0000VJ-U1 for mharc-autoconf-patches@gnu.org; Tue, 06 May 2008 18:01:17 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JtVDo-0000V9-O4 for autoconf-patches@gnu.org; Tue, 06 May 2008 18:01:16 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JtVDn-0000Um-0H for autoconf-patches@gnu.org; Tue, 06 May 2008 18:01:15 -0400 Received: from [199.232.76.173] (port=35318 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JtVDm-0000Uj-RP for autoconf-patches@gnu.org; Tue, 06 May 2008 18:01:14 -0400 Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1JtVDm-0005GH-Gk for autoconf-patches@gnu.org; Tue, 06 May 2008 18:01:14 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JtVDg-0005fW-Kx for autoconf-patches@gnu.org; Tue, 06 May 2008 22:01:09 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 06 May 2008 22:01:08 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 06 May 2008 22:01:08 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Tue, 6 May 2008 22:00:59 +0000 (UTC) Lines: 16 Message-ID: References: <20080506204153.GC16385@ins.uni-bonn.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Re: let AC_MSG_FAILURE report pwd X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2008 22:01:16 -0000 Ralf Wildenhues gmx.de> writes: > People simply don't realize that there could be more than one config.log > and config.cache file. Myself included - I've been bitten on this in the past when building cygwin, which shares the same ubertree multiple config.cache issue with gcc. > > OK to apply? Yes please. -- Eric Blake From MAILER-DAEMON Sat May 10 11:37:41 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1Jur8n-0006wY-1A for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 11:37:41 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Jur8l-0006wF-Kn for autoconf-patches@gnu.org; Sat, 10 May 2008 11:37:39 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Jur8j-0006vq-Ul for autoconf-patches@gnu.org; Sat, 10 May 2008 11:37:39 -0400 Received: from [199.232.76.173] (port=54069 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Jur8j-0006vn-OY for autoconf-patches@gnu.org; Sat, 10 May 2008 11:37:37 -0400 Received: from [83.143.18.155] (port=48676 helo=barvaux.org) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Jur8i-0008Ib-JI for autoconf-patches@gnu.org; Sat, 10 May 2008 11:37:37 -0400 Received: (qmail 26768 invoked by uid 210); 10 May 2008 15:37:23 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr by isfet (envelope-from , uid 201) with qmail-scanner-2.02st (clamdscan: 0.93/7081. spamassassin: 3.2.1. perlscan: 2.02st. Clear:RC:1(193.251.67.168):. Processed in 0.036878 secs); 10 May 2008 15:37:23 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr (HELO shou.barvaux.org) (193.251.67.168) by 83.143.18.155 with ESMTPS (DHE-RSA-AES256-SHA encrypted); 10 May 2008 15:37:23 -0000 Date: Sat, 10 May 2008 17:37:25 +0200 From: Didier Barvaux To: autoconf-patches@gnu.org Message-ID: <20080510173725.054b971a@shou.barvaux.org> X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.8; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/0c2OFdu6Nzi+7s6ZvAacEO7"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) Subject: fix AC_C_CONST to work with CFLAGS -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 15:37:39 -0000 --Sig_/0c2OFdu6Nzi+7s6ZvAacEO7 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi all, I seems that GCC fails to compile the code generated by the AC_C_CONST macro if the -O2 -Wall -Werror CFLAGS are specified. I tried with both autoconf 2.61 and 2.62.17-684d2-dirty (taken from git repo). GCC version is 4.1.2. Below is the part of the config.log file describing what appends: configure:25803: checking for an ANSI C-conforming const configure:25878: gcc -c -g -O2 -Wno-unused-function -Wall -Werror conftest.= c >&5 cc1: warnings being treated as errors conftest.c: In function 'main': conftest.c:76: warning: 't' is used uninitialized in this function conftest.c:92: warning: 'b' is used uninitialized in this function conftest.c:98: warning: 'cs[0]' is used uninitialized in this function configure:25885: $? =3D 1 configure: failed program was: [...] | int | main () | { | /* FIXME: Include the comments suggested by Paul. */ | #ifndef __cplusplus | /* Ultrix mips cc rejects this. */ | typedef int charset[2]; | const charset cs; | /* SunOS 4.1.1 cc rejects this. */ | char const *const *pcpcc; | char **ppc; | /* NEC SVR4.0.2 mips cc rejects this. */ | struct point {int x, y;}; | static struct point const zero =3D {0,0}; | /* AIX XL C 1.02.0.0 rejects this. | It does not let you subtract one const X* pointer from another in | an arm of an if-expression whose if-part is not a constant | expression */ | const char *g =3D "string"; | pcpcc =3D &g + (g ? g-g : 0); | /* HPUX 7.0 cc rejects these. */ | ++pcpcc; | ppc =3D (char**) pcpcc; | pcpcc =3D (char const *const *) ppc; | { /* SCO 3.2v4 cc rejects this. */ | char *t; | char const *s =3D 0 ? (char *) 0 : (char const *) 0; |=20 | *t++ =3D 0; | if (s) return 0; | } | { /* Someone thinks the Sun supposedly-ANSI compiler will reject this. = */ | int x[] =3D {25, 17}; | const int *foo =3D &x[0]; | ++foo; | } | { /* Sun SC1.0 ANSI compiler rejects this -- but not the above. */ | typedef const int *iptr; | iptr p =3D 0; | ++p; | } | { /* AIX XL C 1.02.0.0 rejects this saying | "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue= . */ | struct s { int j; const int *ap[3]; }; | struct s *b; b->j =3D 5; | } | { /* ULTRIX-32 V3.1 (Rev 9) vcc rejects this */ | const int foo =3D 10; | if (!foo) return 0; | } | return !cs[0] && !zero.x; | #endif |=20 | ; | return 0; | } configure:25900: result: no Below is a patch that corrects the problem with AC_C_CONST. This patch works great on my system. It may perhaps break the macro on other systems, I don't know. diff --git a/lib/autoconf/c.m4 b/lib/autoconf/c.m4 index ffaa06e..75b0745 100644 --- a/lib/autoconf/c.m4 +++ b/lib/autoconf/c.m4 @@ -1587,7 +1587,7 @@ AC_DEFUN([AC_C_CONST], #ifndef __cplusplus /* Ultrix mips cc rejects this. */ typedef int charset[2]; - const charset cs; + const charset cs =3D { 0, 0 }; /* SunOS 4.1.1 cc rejects this. */ char const *const *pcpcc; char **ppc; @@ -1605,10 +1605,7 @@ AC_DEFUN([AC_C_CONST], ppc =3D (char**) pcpcc; pcpcc =3D (char const *const *) ppc; { /* SCO 3.2v4 cc rejects this. */ - char *t; char const *s =3D 0 ? (char *) 0 : (char const *) 0; - - *t++ =3D 0; if (s) return 0; } { /* Someone thinks the Sun supposedly-ANSI compiler will reject this. = */ @@ -1624,7 +1621,9 @@ AC_DEFUN([AC_C_CONST], { /* AIX XL C 1.02.0.0 rejects this saying "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue.= */ struct s { int j; const int *ap[3]; }; - struct s *b; b->j =3D 5; + struct s a; + struct s *b =3D &a; + b->j =3D 5; } { /* ULTRIX-32 V3.1 (Rev 9) vcc rejects this */ const int foo =3D 10; Any comments are welcome. Didier Barvaux --Sig_/0c2OFdu6Nzi+7s6ZvAacEO7 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkglwT0ACgkQhK2QMwCOja12zgCeJfn0iomZPp661+M6C3Gf7hqB AaMAoLeoeaEHrP/kyp6g7pmJc+BTLE6A =hJev -----END PGP SIGNATURE----- --Sig_/0c2OFdu6Nzi+7s6ZvAacEO7-- From MAILER-DAEMON Sat May 10 11:38:18 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1Jur9O-00071p-QT for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 11:38:18 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Jur9M-00071X-Kl for autoconf-patches@gnu.org; Sat, 10 May 2008 11:38:16 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Jur9L-00071I-8s for autoconf-patches@gnu.org; Sat, 10 May 2008 11:38:15 -0400 Received: from [199.232.76.173] (port=54086 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Jur9L-00071F-4A for autoconf-patches@gnu.org; Sat, 10 May 2008 11:38:15 -0400 Received: from [83.143.18.155] (port=48678 helo=barvaux.org) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Jur9K-0008PT-Io for autoconf-patches@gnu.org; Sat, 10 May 2008 11:38:14 -0400 Received: (qmail 26799 invoked by uid 210); 10 May 2008 15:38:01 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr by isfet (envelope-from , uid 201) with qmail-scanner-2.02st (clamdscan: 0.93/7081. spamassassin: 3.2.1. perlscan: 2.02st. Clear:RC:1(193.251.67.168):. Processed in 0.033993 secs); 10 May 2008 15:38:01 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr (HELO shou.barvaux.org) (193.251.67.168) by 83.143.18.155 with ESMTPS (DHE-RSA-AES256-SHA encrypted); 10 May 2008 15:38:01 -0000 Date: Sat, 10 May 2008 17:37:57 +0200 From: Didier Barvaux To: autoconf-patches@gnu.org Message-ID: <20080510173757.551c1bf9@shou.barvaux.org> X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.8; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/hDq_9Bzbsu1nrYlePYdSc/R"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) Subject: Extend AT_CHECK_MACRO to test macros with -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 15:38:16 -0000 --Sig_/hDq_9Bzbsu1nrYlePYdSc/R Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi all, I encountered a problem with the AC_C_CONST macro when using -O2 -Wall -Werror as CFLAGS. The AC_C_CONST macro broke with similar CFLAGS some time ago [1]. Thus I created a patch that extends the AT_CHECK_MACRO macro used in the testsuite. [1] http://lists.gnu.org/archive/html/bug-autoconf/2006-07/msg00011.html With the patch, the AT_CHECK_MACRO macro checks that the tested macro (such as the AC_C_CONST macro) is also run with the CFLAGS variable set to "-O2 -Wall -Werror". The macro must give the same result with or without the CFLAGS, otherwise the test fails. This should avoid a similar bug to appear :) diff --git a/tests/local.at b/tests/local.at index a772a97..aa1f543 100644 --- a/tests/local.at +++ b/tests/local.at @@ -427,6 +427,9 @@ AT_CHECK_ENV # We run `configure' twice, both times with a cache, and compare # the environment after each run to detect inconsistencies. # +# We run `configure' one more time with CFLAGS to check that unused or +# uninitialized variables do not change the macro result. +# # New macros are not expected to depend upon obsolete macros. m4_define([AT_CHECK_MACRO], [AT_SETUP([$1]) @@ -447,6 +450,20 @@ done AT_CMP([config-h.r1], [config-h.r2]) AT_CONFIG_CMP([state-env.r1], [state-env.r2]) =20 +AT_CONFIGURE_AC([CFLAGS=3D"-O2 -Wall -Werror -fno-builtin" +m4_default([$2], [$1])]) + +AT_CHECK_AUTOCONF([m4_default([$4], [-W obsolete])]) +AT_CHECK_AUTOHEADER + +AT_CHECK_CONFIGURE([]) +cp -f state-env.after state-env.r3 +cp -f config.h config-h.r3 +AT_CHECK_ENV + +AT_CMP([config-h.r1], [config-h.r3]) +AT_CONFIG_CMP([state-env.r1], [state-env.r3]) + $3 =20 AT_CLEANUP[]dnl Any comments are welcome. Didier Barvaux --Sig_/hDq_9Bzbsu1nrYlePYdSc/R Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkglwWMACgkQhK2QMwCOja1KlQCgjNG3RyqYtlPOdzfzKnCzh7c4 XcIAn3oi0elJNgto8bA534qXoWcVuFW6 =aPSX -----END PGP SIGNATURE----- --Sig_/hDq_9Bzbsu1nrYlePYdSc/R-- From MAILER-DAEMON Sat May 10 12:12:01 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1Jurg1-0003vS-Nw for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 12:12:01 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Jurg0-0003uo-5C for autoconf-patches@gnu.org; Sat, 10 May 2008 12:12:00 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Jurfz-0003uJ-9E for autoconf-patches@gnu.org; Sat, 10 May 2008 12:11:59 -0400 Received: from [199.232.76.173] (port=40116 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Jurfy-0003uD-Vx for autoconf-patches@gnu.org; Sat, 10 May 2008 12:11:59 -0400 Received: from qmta02.emeryville.ca.mail.comcast.net ([76.96.30.24]:40114) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1Jurfy-0005Ns-HH for autoconf-patches@gnu.org; Sat, 10 May 2008 12:11:58 -0400 Received: from OMTA04.emeryville.ca.mail.comcast.net ([76.96.30.35]) by QMTA02.emeryville.ca.mail.comcast.net with comcast id PnsS1Z0030lTkoCA20EQ00; Sat, 10 May 2008 16:09:45 +0000 Received: from [192.168.0.103] ([67.166.125.73]) by OMTA04.emeryville.ca.mail.comcast.net with comcast id PsBp1Z0061b8C2B8Q00000; Sat, 10 May 2008 16:11:50 +0000 X-Authority-Analysis: v=1.0 c=1 a=xJxCvt3Ppr8A:10 a=5C29kbkctMQA:10 a=xe8BsctaAAAA:8 a=AVUQ2IET6mCwywHUV-0A:9 a=ON-iLwrmAZGU-eba687CkHoIXN4A:4 a=eDFNAWYWrCwA:10 a=rPt6xJ-oxjAA:10 Message-ID: <4825C952.1070408@byu.net> Date: Sat, 10 May 2008 10:12:02 -0600 From: Eric Blake User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080421 Thunderbird/2.0.0.14 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: Didier Barvaux References: <20080510173725.054b971a@shou.barvaux.org> In-Reply-To: <20080510173725.054b971a@shou.barvaux.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized. Cc: autoconf-patches@gnu.org Subject: Re: fix AC_C_CONST to work with CFLAGS -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 16:12:00 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 According to Didier Barvaux on 5/10/2008 9:37 AM: | Hi all, | | I seems that GCC fails to compile the code generated by the AC_C_CONST | macro if the -O2 -Wall -Werror CFLAGS are specified. Thanks for the report. AC_C_CONST is obsolete - as of autoconf 2.60, there were no known compilers on typical porting platforms that lacked const support. The easier fix is just to delete that macro from gcc's configure.ac, and require C89 support from the system compiler. And trying to bootstrap a package with -Werror is so prone to failure that in general, we don't guarantee that is possible (even though some packages go out of their way to try to support it). | | Below is a patch that corrects the problem with AC_C_CONST. This patch | works great on my system. It may perhaps break the macro on other | systems, I don't know. | | #ifndef __cplusplus | /* Ultrix mips cc rejects this. */ | typedef int charset[2]; | - const charset cs; | + const charset cs = { 0, 0 }; Changing this looks fishy. | @@ -1605,10 +1605,7 @@ AC_DEFUN([AC_C_CONST], | ppc = (char**) pcpcc; | pcpcc = (char const *const *) ppc; | { /* SCO 3.2v4 cc rejects this. */ | - char *t; | char const *s = 0 ? (char *) 0 : (char const *) 0; | - | - *t++ = 0; | if (s) return 0; Seems reasonable. | @@ -1624,7 +1621,9 @@ AC_DEFUN([AC_C_CONST], | { /* AIX XL C 1.02.0.0 rejects this saying | "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue. */ | struct s { int j; const int *ap[3]; }; | - struct s *b; b->j = 5; | + struct s a; | + struct s *b = &a; | + b->j = 5; | } Also seems reasonable. | | Any comments are welcome. I'd like to hear from at least Paul Eggert before applying (part or all of) this, as he has more experience with AC_C_CONST. - -- Don't work too hard, make some time for fun as well! Eric Blake ebb9@byu.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkglyVEACgkQ84KuGfSFAYD8dwCZAVF4pR8i1MNzf2hKaj3s0Tl2 apgAn1Im7ycQZA0Ycsr6ufHztPzAA+6d =up+P -----END PGP SIGNATURE----- From MAILER-DAEMON Sat May 10 12:15:38 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JurjW-0004c8-ON for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 12:15:38 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JurjU-0004bl-FZ for autoconf-patches@gnu.org; Sat, 10 May 2008 12:15:36 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JurjT-0004bZ-1s for autoconf-patches@gnu.org; Sat, 10 May 2008 12:15:35 -0400 Received: from [199.232.76.173] (port=37278 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JurjS-0004bW-U9 for autoconf-patches@gnu.org; Sat, 10 May 2008 12:15:34 -0400 Received: from qmta05.westchester.pa.mail.comcast.net ([76.96.62.48]:43145) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JurjS-0005tU-H2 for autoconf-patches@gnu.org; Sat, 10 May 2008 12:15:34 -0400 Received: from OMTA14.westchester.pa.mail.comcast.net ([76.96.62.60]) by QMTA05.westchester.pa.mail.comcast.net with comcast id Pqwj1Z0041HzFnQ5504M00; Sat, 10 May 2008 16:14:58 +0000 Received: from [192.168.0.103] ([67.166.125.73]) by OMTA14.westchester.pa.mail.comcast.net with comcast id PsFY1Z0041b8C2B3a00000; Sat, 10 May 2008 16:15:34 +0000 X-Authority-Analysis: v=1.0 c=1 a=SOhZI_TSfSgA:10 a=QLMy2U4EIWAA:10 a=xe8BsctaAAAA:8 a=IIbjWCJtrmUsPBsDhH8A:9 a=w3dZU2irLW5tJCELZ0mYz92wNBgA:4 a=eDFNAWYWrCwA:10 a=rPt6xJ-oxjAA:10 Message-ID: <4825CA30.2070405@byu.net> Date: Sat, 10 May 2008 10:15:44 -0600 From: Eric Blake User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080421 Thunderbird/2.0.0.14 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: Didier Barvaux References: <20080510173757.551c1bf9@shou.barvaux.org> In-Reply-To: <20080510173757.551c1bf9@shou.barvaux.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized. Cc: autoconf-patches@gnu.org Subject: Re: Extend AT_CHECK_MACRO to test macros with -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 16:15:37 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 According to Didier Barvaux on 5/10/2008 9:37 AM: | Hi all, | | I encountered a problem with the AC_C_CONST macro when using -O2 -Wall | -Werror as CFLAGS. The AC_C_CONST macro broke with similar CFLAGS some | time ago [1]. Thus I created a patch that extends the AT_CHECK_MACRO | macro used in the testsuite. Thanks for the patch. However, I don't want to apply it just yet. There is a bigger issue that several tests fail when running the autoconf testsuite under -Wall -Werror (I have tried this in the past); in particular, some warnings occur when testing for functions that might be implemented as a compiler builtin (such as printf) for which I don't know how to silence the warning. Changing AT_CHECK_MACRO without a fix for every macro provided by autoconf will lead to false testsuite failures. - -- Don't work too hard, make some time for fun as well! Eric Blake ebb9@byu.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkglyjAACgkQ84KuGfSFAYCegQCfX3IgUoMTxx+9a+HaDLsJzimR TDoAoMKsmTg+ffGd4guxDS2AVqSjj20x =fJAT -----END PGP SIGNATURE----- From MAILER-DAEMON Sat May 10 14:13:04 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JutZA-0002Gg-Cy for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 14:13:04 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JuojC-0007Y2-FK for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:06 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Juoj9-0007Xq-Vt for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:05 -0400 Received: from [199.232.76.173] (port=59101 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Juoj9-0007Xn-RW for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:03 -0400 Received: from [83.143.18.155] (port=56430 helo=barvaux.org) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Juoj8-0007ud-Eh for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:03 -0400 Received: (qmail 28845 invoked by uid 210); 10 May 2008 12:36:07 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr by isfet (envelope-from , uid 201) with qmail-scanner-2.02st (clamdscan: 0.93/7081. spamassassin: 3.2.1. perlscan: 2.02st. Clear:RC:1(193.251.67.168):. Processed in 0.036622 secs); 10 May 2008 12:36:07 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr (HELO shou.barvaux.org) (193.251.67.168) by 83.143.18.155 with ESMTPS (DHE-RSA-AES256-SHA encrypted); 10 May 2008 12:36:06 -0000 Date: Sat, 10 May 2008 14:36:01 +0200 From: Didier Barvaux To: autoconf-patches@gnu.org Message-ID: <20080510143601.3867400e@shou.barvaux.org> X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.8; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/uPFyTaysL/kzNlyE6HE+xTf"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) X-Greylist: delayed 1601 seconds by postgrey-1.27 at monty-python; Sat, 10 May 2008 09:03:00 EDT X-Mailman-Approved-At: Sat, 10 May 2008 14:13:02 -0400 Subject: fix AC_C_CONST to work with CFLAGS -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 13:03:07 -0000 --Sig_/uPFyTaysL/kzNlyE6HE+xTf Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi all, I seems that GCC fails to compile the code generated by the AC_C_CONST macro if the -O2 -Wall -Werror CFLAGS are specified. I tried with both autoconf 2.61 and 2.62.17-684d2-dirty (taken from git repo). GCC version is 4.1.2. Below is the part of the config.log file describing what appends: configure:25803: checking for an ANSI C-conforming const configure:25878: gcc -c -g -O2 -Wno-unused-function -Wall -Werror conftest.= c >&5 cc1: warnings being treated as errors conftest.c: In function 'main': conftest.c:76: warning: 't' is used uninitialized in this function conftest.c:92: warning: 'b' is used uninitialized in this function conftest.c:98: warning: 'cs[0]' is used uninitialized in this function configure:25885: $? =3D 1 configure: failed program was: [...] | int | main () | { | /* FIXME: Include the comments suggested by Paul. */ | #ifndef __cplusplus | /* Ultrix mips cc rejects this. */ | typedef int charset[2]; | const charset cs; | /* SunOS 4.1.1 cc rejects this. */ | char const *const *pcpcc; | char **ppc; | /* NEC SVR4.0.2 mips cc rejects this. */ | struct point {int x, y;}; | static struct point const zero =3D {0,0}; | /* AIX XL C 1.02.0.0 rejects this. | It does not let you subtract one const X* pointer from another in | an arm of an if-expression whose if-part is not a constant | expression */ | const char *g =3D "string"; | pcpcc =3D &g + (g ? g-g : 0); | /* HPUX 7.0 cc rejects these. */ | ++pcpcc; | ppc =3D (char**) pcpcc; | pcpcc =3D (char const *const *) ppc; | { /* SCO 3.2v4 cc rejects this. */ | char *t; | char const *s =3D 0 ? (char *) 0 : (char const *) 0; |=20 | *t++ =3D 0; | if (s) return 0; | } | { /* Someone thinks the Sun supposedly-ANSI compiler will reject this. = */ | int x[] =3D {25, 17}; | const int *foo =3D &x[0]; | ++foo; | } | { /* Sun SC1.0 ANSI compiler rejects this -- but not the above. */ | typedef const int *iptr; | iptr p =3D 0; | ++p; | } | { /* AIX XL C 1.02.0.0 rejects this saying | "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue= . */ | struct s { int j; const int *ap[3]; }; | struct s *b; b->j =3D 5; | } | { /* ULTRIX-32 V3.1 (Rev 9) vcc rejects this */ | const int foo =3D 10; | if (!foo) return 0; | } | return !cs[0] && !zero.x; | #endif |=20 | ; | return 0; | } configure:25900: result: no Below is a patch that corrects the problem with AC_C_CONST. This patch works great on my system, but it may break the macro on other systems. diff --git a/lib/autoconf/c.m4 b/lib/autoconf/c.m4 index ffaa06e..75b0745 100644 --- a/lib/autoconf/c.m4 +++ b/lib/autoconf/c.m4 @@ -1587,7 +1587,7 @@ AC_DEFUN([AC_C_CONST], #ifndef __cplusplus /* Ultrix mips cc rejects this. */ typedef int charset[2]; - const charset cs; + const charset cs =3D { 0, 0 }; /* SunOS 4.1.1 cc rejects this. */ char const *const *pcpcc; char **ppc; @@ -1605,10 +1605,7 @@ AC_DEFUN([AC_C_CONST], ppc =3D (char**) pcpcc; pcpcc =3D (char const *const *) ppc; { /* SCO 3.2v4 cc rejects this. */ - char *t; char const *s =3D 0 ? (char *) 0 : (char const *) 0; - - *t++ =3D 0; if (s) return 0; } { /* Someone thinks the Sun supposedly-ANSI compiler will reject this. = */ @@ -1624,7 +1621,9 @@ AC_DEFUN([AC_C_CONST], { /* AIX XL C 1.02.0.0 rejects this saying "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue.= */ struct s { int j; const int *ap[3]; }; - struct s *b; b->j =3D 5; + struct s a; + struct s *b =3D &a; + b->j =3D 5; } { /* ULTRIX-32 V3.1 (Rev 9) vcc rejects this */ const int foo =3D 10; Any comments are welcome. Didier Barvaux --Sig_/uPFyTaysL/kzNlyE6HE+xTf Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkgllr8ACgkQhK2QMwCOja3OuwCeNO9J6YzS4QP7aNsT1BO4OWvN k/cAnil6Qtx4V9oj1ZeIY7rHzDq7a6rF =LUOG -----END PGP SIGNATURE----- --Sig_/uPFyTaysL/kzNlyE6HE+xTf-- From MAILER-DAEMON Sat May 10 14:14:11 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JutaF-0002s9-Kh for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 14:14:11 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JuojV-0007gU-Ny for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:25 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JuojU-0007gF-Q3 for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:24 -0400 Received: from [199.232.76.173] (port=59119 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JuojU-0007gC-H8 for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:24 -0400 Received: from [83.143.18.155] (port=56431 helo=barvaux.org) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1JuojT-0007z5-Qt for autoconf-patches@gnu.org; Sat, 10 May 2008 09:03:24 -0400 Received: (qmail 28857 invoked by uid 210); 10 May 2008 12:36:29 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr by isfet (envelope-from , uid 201) with qmail-scanner-2.02st (clamdscan: 0.93/7081. spamassassin: 3.2.1. perlscan: 2.02st. Clear:RC:1(193.251.67.168):. Processed in 0.033712 secs); 10 May 2008 12:36:29 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr (HELO shou.barvaux.org) (193.251.67.168) by 83.143.18.155 with ESMTPS (DHE-RSA-AES256-SHA encrypted); 10 May 2008 12:36:29 -0000 Date: Sat, 10 May 2008 14:36:25 +0200 From: Didier Barvaux To: autoconf-patches@gnu.org Message-ID: <20080510143625.6c2e3d20@shou.barvaux.org> X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.8; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/NRGZgVxfis_k8.EHTul7dQq"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) X-Mailman-Approved-At: Sat, 10 May 2008 14:14:10 -0400 Subject: Extend AT_CHECK_MACRO to test macros with -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 13:03:26 -0000 --Sig_/NRGZgVxfis_k8.EHTul7dQq Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi all, I encountered a problem with the AC_C_CONST macro when using -O2 -Wall -Werror as CFLAGS. The AC_C_CONST macro broke with similar CFLAGS some time ago [1]. Thus I created a patch that extends the AT_CHECK_MACRO macro used in the testsuite. [1] http://lists.gnu.org/archive/html/bug-autoconf/2006-07/msg00011.html With the patch, the AT_CHECK_MACRO macro checks that the tested macro (such as the AC_C_CONST macro) is also run with the CFLAGS variable set to "-O2 -Wall -Werror". The macro must give the same result with or without the CFLAGS, otherwise the test fails. This should avoid a similar bug to appear :) diff --git a/tests/local.at b/tests/local.at index a772a97..aa1f543 100644 --- a/tests/local.at +++ b/tests/local.at @@ -427,6 +427,9 @@ AT_CHECK_ENV # We run `configure' twice, both times with a cache, and compare # the environment after each run to detect inconsistencies. # +# We run `configure' one more time with CFLAGS to check that unused or +# uninitialized variables do not change the macro result. +# # New macros are not expected to depend upon obsolete macros. m4_define([AT_CHECK_MACRO], [AT_SETUP([$1]) @@ -447,6 +450,20 @@ done AT_CMP([config-h.r1], [config-h.r2]) AT_CONFIG_CMP([state-env.r1], [state-env.r2]) =20 +AT_CONFIGURE_AC([CFLAGS=3D"-O2 -Wall -Werror -fno-builtin" +m4_default([$2], [$1])]) + +AT_CHECK_AUTOCONF([m4_default([$4], [-W obsolete])]) +AT_CHECK_AUTOHEADER + +AT_CHECK_CONFIGURE([]) +cp -f state-env.after state-env.r3 +cp -f config.h config-h.r3 +AT_CHECK_ENV + +AT_CMP([config-h.r1], [config-h.r3]) +AT_CONFIG_CMP([state-env.r1], [state-env.r3]) + $3 =20 AT_CLEANUP[]dnl Any comments are welcome. Didier Barvaux --Sig_/NRGZgVxfis_k8.EHTul7dQq Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkglltcACgkQhK2QMwCOja1nMwCgtivXqOnhc2kqffNNHNnA29Ee A14AniFyI/4rRgEiPVyqXPBJqpiN9lFE =J2ge -----END PGP SIGNATURE----- --Sig_/NRGZgVxfis_k8.EHTul7dQq-- From MAILER-DAEMON Sat May 10 14:39:47 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1Jutz0-00031y-UV for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 14:39:46 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Jutyz-00031s-9F for autoconf-patches@gnu.org; Sat, 10 May 2008 14:39:45 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Jutyx-00031f-Rp for autoconf-patches@gnu.org; Sat, 10 May 2008 14:39:44 -0400 Received: from [199.232.76.173] (port=35432 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Jutyx-00031c-LF for autoconf-patches@gnu.org; Sat, 10 May 2008 14:39:43 -0400 Received: from [83.143.18.155] (port=35266 helo=barvaux.org) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Jutyx-0003hN-2n for autoconf-patches@gnu.org; Sat, 10 May 2008 14:39:43 -0400 Received: (qmail 27551 invoked by uid 210); 10 May 2008 18:39:28 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr by isfet (envelope-from , uid 201) with qmail-scanner-2.02st (clamdscan: 0.93/7081. spamassassin: 3.2.1. perlscan: 2.02st. Clear:RC:1(193.251.67.168):. Processed in 0.031971 secs); 10 May 2008 18:39:28 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr (HELO shou.barvaux.org) (193.251.67.168) by 83.143.18.155 with ESMTPS (DHE-RSA-AES256-SHA encrypted); 10 May 2008 18:39:28 -0000 Date: Sat, 10 May 2008 20:39:16 +0200 From: Didier Barvaux To: autoconf-patches@gnu.org Message-ID: <20080510203916.105e91c8@shou.barvaux.org> In-Reply-To: <4825CA30.2070405@byu.net> References: <20080510173757.551c1bf9@shou.barvaux.org> <4825CA30.2070405@byu.net> X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.8; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/1D5=gWLgAgIXhucZEHp3kWd"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) Subject: Re: Extend AT_CHECK_MACRO to test macros with -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 18:39:45 -0000 --Sig_/1D5=gWLgAgIXhucZEHp3kWd Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable > Thanks for the patch. However, I don't want to apply it just yet. > There is a bigger issue that several tests fail when running the > autoconf testsuite under -Wall -Werror (I have tried this in the > past); in particular, some warnings occur when testing for functions > that might be implemented as a compiler builtin (such as printf) for > which I don't know how to silence the warning. Changing > AT_CHECK_MACRO without a fix for every macro provided by autoconf > will lead to false testsuite failures. Look at the patch. Use -fno-builtin to silence the warning. On my system, the testsuite gives me the same results with the patches applied (the one for AC_C_CONST and the one for AT_CHECK_MACRO) and without. However some tests fails in both cases. Didier --Sig_/1D5=gWLgAgIXhucZEHp3kWd Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkgl6+oACgkQhK2QMwCOja27rACgqJh4TcPn0uLGPXRdD+ppay/l NWEAoLyYVfWxOEZ3QXLnwYIgo4t8BYs3 =MUbZ -----END PGP SIGNATURE----- --Sig_/1D5=gWLgAgIXhucZEHp3kWd-- From MAILER-DAEMON Sat May 10 14:42:19 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1Juu1S-0003rR-St for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 14:42:18 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Juu1R-0003r9-Qd for autoconf-patches@gnu.org; Sat, 10 May 2008 14:42:17 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Juu1R-0003qn-6Q for autoconf-patches@gnu.org; Sat, 10 May 2008 14:42:17 -0400 Received: from [199.232.76.173] (port=35499 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Juu1Q-0003qf-VC for autoconf-patches@gnu.org; Sat, 10 May 2008 14:42:16 -0400 Received: from [83.143.18.155] (port=35268 helo=barvaux.org) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Juu1Q-00043h-FG for autoconf-patches@gnu.org; Sat, 10 May 2008 14:42:16 -0400 Received: (qmail 27595 invoked by uid 210); 10 May 2008 18:42:02 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr by isfet (envelope-from , uid 201) with qmail-scanner-2.02st (clamdscan: 0.93/7081. spamassassin: 3.2.1. perlscan: 2.02st. Clear:RC:1(193.251.67.168):. Processed in 0.03314 secs); 10 May 2008 18:42:02 -0000 Received: from laubervilliers-151-11-35-168.w193-251.abo.wanadoo.fr (HELO shou.barvaux.org) (193.251.67.168) by 83.143.18.155 with ESMTPS (DHE-RSA-AES256-SHA encrypted); 10 May 2008 18:42:02 -0000 Date: Sat, 10 May 2008 20:42:04 +0200 From: Didier Barvaux To: autoconf-patches@gnu.org Message-ID: <20080510204204.43a912b7@shou.barvaux.org> In-Reply-To: <4825C952.1070408@byu.net> References: <20080510173725.054b971a@shou.barvaux.org> <4825C952.1070408@byu.net> X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.8; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/AKtjWjU0T6aN5zlq0Slm9Ak"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) Subject: Re: fix AC_C_CONST to work with CFLAGS -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 18:42:18 -0000 --Sig_/AKtjWjU0T6aN5zlq0Slm9Ak Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable > Thanks for the report. AC_C_CONST is obsolete - as of autoconf 2.60, > there were no known compilers on typical porting platforms that lacked > const support. The easier fix is just to delete that macro from gcc's > configure.ac, and require C89 support from the system compiler. Don't known about AC_C_CONST being obsolete. No message is printed during configure to warn about. Didier Barvaux --Sig_/AKtjWjU0T6aN5zlq0Slm9Ak Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkgl7IQACgkQhK2QMwCOja1bwwCdGyIvt9Q7Y7Od92BqZNLb8tFF sigAoK7HbNpQx/1TtXUcbpl+2hahPe1t =O9bP -----END PGP SIGNATURE----- --Sig_/AKtjWjU0T6aN5zlq0Slm9Ak-- From MAILER-DAEMON Sat May 10 14:54:30 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JuuDG-0000sP-3c for mharc-autoconf-patches@gnu.org; Sat, 10 May 2008 14:54:30 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JuuDE-0000rH-5n for autoconf-patches@gnu.org; Sat, 10 May 2008 14:54:28 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JuuD9-0000oT-Iz for autoconf-patches@gnu.org; Sat, 10 May 2008 14:54:24 -0400 Received: from [199.232.76.173] (port=43786 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JuuD9-0000oM-9t for autoconf-patches@gnu.org; Sat, 10 May 2008 14:54:23 -0400 Received: from qmta01.westchester.pa.mail.comcast.net ([76.96.62.16]:49894) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JuuD9-0005lp-2L for autoconf-patches@gnu.org; Sat, 10 May 2008 14:54:23 -0400 Received: from OMTA09.westchester.pa.mail.comcast.net ([76.96.62.20]) by QMTA01.westchester.pa.mail.comcast.net with comcast id PrA51Z0090SCNGk510Dk00; Sat, 10 May 2008 18:54:21 +0000 Received: from [192.168.0.103] ([67.166.125.73]) by OMTA09.westchester.pa.mail.comcast.net with comcast id PuuK1Z0051b8C2B3V00000; Sat, 10 May 2008 18:54:21 +0000 X-Authority-Analysis: v=1.0 c=1 a=SOhZI_TSfSgA:10 a=QLMy2U4EIWAA:10 a=xe8BsctaAAAA:8 a=m9bWWF7hjKz3YKDj8mkA:9 a=ryBc8DOVqGWrMXUZbZE5Vl9WOEkA:4 a=eDFNAWYWrCwA:10 a=rPt6xJ-oxjAA:10 Message-ID: <4825EF68.3080902@byu.net> Date: Sat, 10 May 2008 12:54:32 -0600 From: Eric Blake User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080421 Thunderbird/2.0.0.14 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: Didier Barvaux References: <20080510173757.551c1bf9@shou.barvaux.org> <4825CA30.2070405@byu.net> <20080510203916.105e91c8@shou.barvaux.org> In-Reply-To: <20080510203916.105e91c8@shou.barvaux.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized. Cc: autoconf-patches@gnu.org Subject: Re: Extend AT_CHECK_MACRO to test macros with -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 May 2008 18:54:28 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 According to Didier Barvaux on 5/10/2008 12:39 PM: | | Look at the patch. Use -fno-builtin to silence the warning. gcc isn't the only compiler, but -fno-builtin is not universally available. Blindly setting CFLAGS to include options that are not globally portable is insufficient; the patch must also make sure that the options are supported by the current compiler. That's where things get tricky, which is why I haven't bothered to try patching it. It also means that any patch to get it working will probably require copyright assignment; would you be interested in starting that process? | | On my system, the testsuite gives me the same results with the patches | applied (the one for AC_C_CONST and the one for AT_CHECK_MACRO) and | without. However some tests fails in both cases. Which tests? Can you post the testsuite.log from an unpatched run? - -- Don't work too hard, make some time for fun as well! Eric Blake ebb9@byu.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkgl72gACgkQ84KuGfSFAYChhQCfXKKy149G6ElSC1rjHb3EnclX 8xQAoLe0ihrj6UzbEkIosgOVKRWPBp9H =OpYE -----END PGP SIGNATURE----- From MAILER-DAEMON Mon May 12 08:46:01 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JvXPl-0000Xo-0X for mharc-autoconf-patches@gnu.org; Mon, 12 May 2008 08:46:01 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JvXPj-0000XQ-KI for autoconf-patches@gnu.org; Mon, 12 May 2008 08:45:59 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JvXPi-0000Wm-44 for autoconf-patches@gnu.org; Mon, 12 May 2008 08:45:58 -0400 Received: from [199.232.76.173] (port=49726 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JvXPi-0000Wi-07 for autoconf-patches@gnu.org; Mon, 12 May 2008 08:45:58 -0400 Received: from qmta01.emeryville.ca.mail.comcast.net ([76.96.30.16]:38670) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JvXPg-0005KJ-PT for autoconf-patches@gnu.org; Mon, 12 May 2008 08:45:57 -0400 Received: from OMTA01.emeryville.ca.mail.comcast.net ([76.96.30.11]) by QMTA01.emeryville.ca.mail.comcast.net with comcast id QcRU1Z0090EPchoA104e00; Mon, 12 May 2008 12:45:55 +0000 Received: from [192.168.0.103] ([67.166.125.73]) by OMTA01.emeryville.ca.mail.comcast.net with comcast id Qclu1Z00L1b8C2B8M00000; Mon, 12 May 2008 12:45:55 +0000 X-Authority-Analysis: v=1.0 c=1 a=vduJnDmscCEA:10 a=_78mMnr-P0UA:10 a=xe8BsctaAAAA:8 a=Uaq5Eo4uH4XsHMbbZbYA:9 a=W8_RdTU6Zw6GNhFwSKgA:7 a=bpz7zWTMf7j0VwGZ4FCX6KRANpQA:4 a=eDFNAWYWrCwA:10 a=rPt6xJ-oxjAA:10 Message-ID: <48283C15.5000505@byu.net> Date: Mon, 12 May 2008 06:46:13 -0600 From: Eric Blake User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080421 Thunderbird/2.0.0.14 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: Keith Marshall References: <200805092116.12370.keithmarshall@users.sourceforge.net> In-Reply-To: <200805092116.12370.keithmarshall@users.sourceforge.net> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized. Cc: Autoconf Patches List , bug-autoconf@gnu.org Subject: Re: autoconf-2.62: `make install' fails if file system is case insensitive X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 12:45:59 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [adding autoconf-patches] According to Keith Marshall on 5/9/2008 2:16 PM: | If autoconf's Makefile is processed by a GNU make-3.81, which has been | configured to respect a case insensitive file system, (IMO, the correct | configuration for use on MS-Windows, Perhaps for MSYS and other native MS-Windows environments, but not for cygwin :) | among other other systems which | also have such file systems), and the build is configured directly | within the source directory, then `make install' fails thus: | | $ make install | Makefile:678: warning: overriding commands for target `install' Thanks for the report and patch. I tested it with BSD make, and it appeared to work just fine, so I'm installing it. This patch is trivial, but you may want to consider assigning copyright to make future patch contribution run more smoothly. Also, your ChangeLog summary took two lines, but git prefers a one-line summary, so I condensed it. - -- Don't work too hard, make some time for fun as well! Eric Blake ebb9@byu.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkgoPBUACgkQ84KuGfSFAYCMzgCfa9AVk6AhHAjThqnZT3clkVPN KDIAnAkUoNAetAdaQCv1yW6MvRUxGv90 =4m79 -----END PGP SIGNATURE----- From MAILER-DAEMON Mon May 12 10:00:02 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JvYZO-0000LX-Io for mharc-autoconf-patches@gnu.org; Mon, 12 May 2008 10:00:02 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JvYZN-0000L9-DX for autoconf-patches@gnu.org; Mon, 12 May 2008 10:00:01 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JvYZK-0000KF-Iy for autoconf-patches@gnu.org; Mon, 12 May 2008 10:00:00 -0400 Received: from [199.232.76.173] (port=60399 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JvYZK-0000KC-DI for autoconf-patches@gnu.org; Mon, 12 May 2008 09:59:58 -0400 Received: from main.gmane.org ([80.91.229.2]:49153 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1JvYZJ-00033p-L5 for autoconf-patches@gnu.org; Mon, 12 May 2008 09:59:57 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JvYZA-0002m2-JL for autoconf-patches@gnu.org; Mon, 12 May 2008 13:59:49 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 May 2008 13:59:48 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 May 2008 13:59:48 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Mon, 12 May 2008 13:59:36 +0000 (UTC) Lines: 16 Message-ID: References: <20080506204153.GC16385@ins.uni-bonn.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Re: let AC_MSG_FAILURE report pwd X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 14:00:02 -0000 Eric Blake byu.net> writes: > > People simply don't realize that there could be more than one config.log > > and config.cache file. > > > > > OK to apply? > > Yes please. I took the liberty of applying it on your behalf. -- Eric Blake From MAILER-DAEMON Mon May 12 10:05:11 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JvYeM-0002Ni-Sw for mharc-autoconf-patches@gnu.org; Mon, 12 May 2008 10:05:10 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JvYeL-0002NF-Sf for autoconf-patches@gnu.org; Mon, 12 May 2008 10:05:09 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JvYeK-0002MV-5h for autoconf-patches@gnu.org; Mon, 12 May 2008 10:05:09 -0400 Received: from [199.232.76.173] (port=47968 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JvYeK-0002MQ-2Z for autoconf-patches@gnu.org; Mon, 12 May 2008 10:05:08 -0400 Received: from main.gmane.org ([80.91.229.2]:46627 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1JvYeI-0003yh-TT for autoconf-patches@gnu.org; Mon, 12 May 2008 10:05:07 -0400 Received: from root by ciao.gmane.org with local (Exim 4.43) id 1JvYeE-00030w-Lz for autoconf-patches@gnu.org; Mon, 12 May 2008 14:05:02 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 May 2008 14:05:02 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 May 2008 14:05:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Mon, 12 May 2008 14:00:35 +0000 (UTC) Lines: 47 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Enforce --help/--version compliance X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 14:05:10 -0000 Since we aren't using automake's gnits option, I figured we could at least use its tests of --help and --version compliance. Committing this: >From e7232d49cab7933f7eca277cf3ab864cd145c5a5 Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Wed, 7 May 2008 09:00:16 -0600 Subject: [PATCH] Enforce --help and --version compliance. * configure.ac (AM_INIT_AUTOMAKE): Add std-options option. Signed-off-by: Eric Blake --- ChangeLog | 5 +++++ configure.ac | 2 +- 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/ChangeLog b/ChangeLog index 41716a5..e4de36e 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,8 @@ +2008-05-12 Eric Blake + + Enforce --help and --version compliance. + * configure.ac (AM_INIT_AUTOMAKE): Add std-options option. + 2008-05-08 Keith Marshall (tiny change) Avoid case-insensitive `make install' vs. `INSTALL' conflict. diff --git a/configure.ac b/configure.ac index f06559e..9fbb235 100644 --- a/configure.ac +++ b/configure.ac @@ -27,7 +27,7 @@ AC_CONFIG_SRCDIR([ChangeLog]) AC_CONFIG_AUX_DIR([build-aux]) -AM_INIT_AUTOMAKE([1.10 dist-bzip2 readme-alpha no-texinfo.tex]) +AM_INIT_AUTOMAKE([1.10 dist-bzip2 readme-alpha no-texinfo.tex std-options]) # We use `/bin/sh -n script' to check that there are no syntax errors # in the scripts. Although incredible, there are /bin/sh that go into -- 1.5.5.1 From MAILER-DAEMON Wed May 14 10:29:37 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JwHz7-00034x-H1 for mharc-autoconf-patches@gnu.org; Wed, 14 May 2008 10:29:37 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JwHz5-00033w-Jw for autoconf-patches@gnu.org; Wed, 14 May 2008 10:29:35 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JwHz1-00032j-0S for autoconf-patches@gnu.org; Wed, 14 May 2008 10:29:35 -0400 Received: from [199.232.76.173] (port=56707 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JwHz0-00032e-IH for autoconf-patches@gnu.org; Wed, 14 May 2008 10:29:30 -0400 Received: from main.gmane.org ([80.91.229.2]:44019 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1JwHz0-0001jh-4i for autoconf-patches@gnu.org; Wed, 14 May 2008 10:29:30 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JwHys-0004FR-QJ for autoconf-patches@gnu.org; Wed, 14 May 2008 14:29:23 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 May 2008 14:29:22 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 May 2008 14:29:22 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Wed, 14 May 2008 14:29:09 +0000 (UTC) Lines: 142 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: more shell portability docs X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 14:29:36 -0000 Does anyone have access to FreeBSD, or know what shell heritage their /bin/sh has? This patch was extracted from reports on the git mailing list against the FreeBSD 6.1 /bin/sh, which I don't have access to. But I reproduced the issues using cygwin's ash, so I'm documenting them. http://marc.info/?l=git&m=121073772610342&w=2 corrupted $? when combining && and | $ sh -c 'true && ! true | true; echo $?' 1 $ ash -c 'true && ! true | true; echo $?' 0 http://marc.info/?l=git&m=121066841020121&w=2 corrupted $? when entering function $ sh -c 'foo(){ echo $1 $?; }; (exit 2); foo $?' 2 2 $ ash -c 'foo(){ echo $1 $?; }; (exit 2); foo $?' 2 0 http://marc.info/?l=git&m=121066841020125&w=2 export of undefined variable creates empty variable $ sh -c 'export foo; env | grep foo' $ ash -c 'export foo; env | grep foo' foo= Of these three, the first two are definite bugs, but for the third, I found the wording in POSIX to be too ambiguous to determine if there was an actual compliance problem. >From 3ca7a51d0fa4e6ed467668503bd0246eb5d275c1 Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Wed, 14 May 2008 08:25:01 -0600 Subject: [PATCH] Document some FreeBSD shell bugs. * doc/autoconf.texi (Limitations of Builtins) : Mention ! issue in compound pipe commands. : Mention difference of exporting an undefined variable. (Shell Functions): Mention loss of $? in entry to shell functions. Extracted from the git mailing list. Signed-off-by: Eric Blake --- ChangeLog | 9 +++++++++ doc/autoconf.texi | 37 +++++++++++++++++++++++++++++++++++-- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index dc02724..3782a5b 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,12 @@ +2008-05-14 Eric Blake + + Document some FreeBSD shell bugs. + * doc/autoconf.texi (Limitations of Builtins) : Mention ! issue + in compound pipe commands. + : Mention difference of exporting an undefined variable. + (Shell Functions): Mention loss of $? in entry to shell functions. + Extracted from the git mailing list. + 2008-05-13 Stepan Kasal Work around MSYS and Cygwin bugs when dealing with trailing space. diff --git a/doc/autoconf.texi b/doc/autoconf.texi index 645cc7b..fd60b7b 100644 --- a/doc/autoconf.texi +++ b/doc/autoconf.texi @@ -12297,7 +12297,7 @@ by all shells! Shell functions are considered portable nowadays, though Autoconf still does not use them (Autotest does). However, some pitfalls have to be -avoided for portable use of shell functions. +avoided for portable use of shell functions (@pxref{Shell Functions}). Some ancient systems have quite small limits on the length of the @samp{#!} line; for instance, 32 @@ -13704,6 +13704,19 @@ subshell if the last command of that subshell was @code {exit} or find a shell that does not exhibit the bug, zsh might be the only shell present on the user's machine. +Likewise, the state of @samp{$?} is not reliable when entering a shell +function. This has the effect that using a function as the first +command in a @command{trap} handler can cause problems. + +@example +$ @kbd{bash -c 'foo()@{ echo $?; @}; trap foo 0; (exit 2); exit 2'; echo $?} +2 +2 +$ @kbd{ash -c 'foo()@{ echo $?; @}; trap foo 0; (exit 2); exit 2'; echo $?} +0 +2 +@end example + Shell variables and functions may share the same namespace, for example with Solaris 10 @command{/bin/sh}: @@ -13762,6 +13775,16 @@ must use @samp{. ./foo}. The Unix version 7 shell did not support negating the exit status of commands with @command{!}, and this feature is still absent from some shells (e.g., Solaris @command{/bin/sh}). +Other shells, such as FreeBSD @command{/bin/sh} or @command{ash}, have +bugs when using @command{!} in compound commands: + +@example +$ @kbd{sh -c 'true && ! true | true; echo $?'} +1 +$ @kbd{ash -c 'true && ! true | true; echo $?'} +0 +@end example + Shell code like this: @example @@ -14138,8 +14161,18 @@ alternately @samp{foo} and @samp{bar}, although they should print only @samp{foo} and then a sequence of @samp{bar}s. Therefore you should @command{export} again each environment variable -that you update. +that you update; the export can occur before or after the assignment. + +Posix is not clear on whether the @command{export} of an undefined +variable causes the variable to be defined with the value of an empty +string, or merely marks any future definition of a variable by that name +for export. Various shells behave differently in this regard: +@example +$ @kbd{sh -c 'export foo; env | grep foo'} +$ @kbd{ash -c 'export foo; env | grep foo'} +foo= +@end example @item @command{false} @c ------------------ -- 1.5.5.1 From MAILER-DAEMON Wed May 14 13:33:56 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JwKrU-0000sS-Dk for mharc-autoconf-patches@gnu.org; Wed, 14 May 2008 13:33:56 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JwKrT-0000qa-02 for autoconf-patches@gnu.org; Wed, 14 May 2008 13:33:55 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JwKrS-0000pb-9I for autoconf-patches@gnu.org; Wed, 14 May 2008 13:33:54 -0400 Received: from [199.232.76.173] (port=60563 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JwKrS-0000pS-5c for autoconf-patches@gnu.org; Wed, 14 May 2008 13:33:54 -0400 Received: from blizzard.cs.caltech.edu ([131.215.44.2]:33925) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JwKrS-0002E7-9L for autoconf-patches@gnu.org; Wed, 14 May 2008 13:33:54 -0400 Received: from localhost (flood.cs.caltech.edu [131.215.44.31]) by blizzard.cs.caltech.edu (Postfix) with ESMTP id CC42B2A3393; Wed, 14 May 2008 10:33:49 -0700 (PDT) Received: from snowstorm.cs.caltech.edu ([131.215.44.21]) by localhost (flood.cs.caltech.edu [131.215.44.31]) (amavisd-new, port 10024) with ESMTP id 27257-07; Wed, 14 May 2008 10:33:49 -0700 (PDT) Received: from orchestra.cs.caltech.edu (orchestra.cs.caltech.edu [131.215.44.20]) by snowstorm.cs.caltech.edu (Postfix) with ESMTP id A621D262FCD; Wed, 14 May 2008 10:33:49 -0700 (PDT) Received: by orchestra.cs.caltech.edu (Postfix, from userid 20310) id 8C9051700B2; Wed, 14 May 2008 10:33:49 -0700 (PDT) Date: Wed, 14 May 2008 10:33:49 -0700 From: Noah Misch To: Eric Blake Message-ID: <20080514173349.GA20375@orchestra.cs.caltech.edu> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.9i X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Cc: autoconf-patches@gnu.org Subject: Re: more shell portability docs X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 17:33:55 -0000 On Wed, May 14, 2008 at 02:29:09PM +0000, Eric Blake wrote: > Does anyone have access to FreeBSD, or know what shell heritage their /bin/sh > has? This patch was extracted from reports on the git mailing list against the > FreeBSD 6.1 /bin/sh, which I don't have access to. But I reproduced the issues > using cygwin's ash, so I'm documenting them. Indeed, ash and {Free,Net}BSD /bin/sh have a common origin. > http://marc.info/?l=git&m=121073772610342&w=2 > corrupted $? when combining && and | > > $ sh -c 'true && ! true | true; echo $?' > 1 > $ ash -c 'true && ! true | true; echo $?' > 0 This bug is unrelated to `&&', affecting any `!'-ed pipeline: $ sh -c '! : | :'; echo $? 1 $ ash -c '! : | :'; echo $? 0 Ash evaluates the pipeline like `{ ! :; } | :'. Also note this bug affecting `!' with a compound command. $ sh -c '! { :; }' $ ash -c '! { :; }' {: not found Syntax error: "}" unexpected From MAILER-DAEMON Wed May 14 15:49:07 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JwMyJ-00022G-7m for mharc-autoconf-patches@gnu.org; Wed, 14 May 2008 15:49:07 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JwMyH-000213-Va for autoconf-patches@gnu.org; Wed, 14 May 2008 15:49:06 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JwMyG-00020Z-V1 for autoconf-patches@gnu.org; Wed, 14 May 2008 15:49:05 -0400 Received: from [199.232.76.173] (port=52065 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JwMyG-00020L-Lw for autoconf-patches@gnu.org; Wed, 14 May 2008 15:49:04 -0400 Received: from main.gmane.org ([80.91.229.2]:49927 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1JwMyG-000329-2S for autoconf-patches@gnu.org; Wed, 14 May 2008 15:49:04 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JwMyD-0004ll-Ro for autoconf-patches@gnu.org; Wed, 14 May 2008 19:49:01 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 May 2008 19:49:01 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 May 2008 19:49:01 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Wed, 14 May 2008 19:48:54 +0000 (UTC) Lines: 86 Message-ID: References: <20080514173349.GA20375@orchestra.cs.caltech.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Re: more shell portability docs X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 19:49:06 -0000 Noah Misch cs.caltech.edu> writes: > > Indeed, ash and {Free,Net}BSD /bin/sh have a common origin. I was a bit surprised to hear that, since /bin/sh OpenBSD 4.0 (the only *BSD system I have access to at the moment) is a pdksh clone instead. Oh well - I learned something today. > > $ sh -c '! : | :'; echo $? > 1 > $ ash -c '! : | :'; echo $? > 0 > > Ash evaluates the pipeline like `{ ! :; } | :'. Also note this bug affecting > `!' with a compound command. > > $ sh -c '! { :; }' > $ ash -c '! { :; }' > {: not found > Syntax error: "}" unexpected Thanks for the clarification. I've updated the manual accordingly (fortunately, the end result is unchanged - portable shell scripts can't use !): >From 95dfc197494a66dc9a88577d18f1633afc7cff2d Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Wed, 14 May 2008 13:41:18 -0600 Subject: [PATCH] Improve documentation of ! issues. * doc/autoconf.texi (Limitations of Builtins) : Touch up. Reported by Noah Misch. Signed-off-by: Eric Blake --- ChangeLog | 4 ++++ doc/autoconf.texi | 12 +++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/ChangeLog b/ChangeLog index 3782a5b..15adde0 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,9 @@ 2008-05-14 Eric Blake + Improve documentation of ! issues. + * doc/autoconf.texi (Limitations of Builtins) : Touch up. + Reported by Noah Misch. + Document some FreeBSD shell bugs. * doc/autoconf.texi (Limitations of Builtins) : Mention ! issue in compound pipe commands. diff --git a/doc/autoconf.texi b/doc/autoconf.texi index fd60b7b..ee1ede5 100644 --- a/doc/autoconf.texi +++ b/doc/autoconf.texi @@ -13776,13 +13776,19 @@ The Unix version 7 shell did not support negating the exit status of commands with @command{!}, and this feature is still absent from some shells (e.g., Solaris @command{/bin/sh}). Other shells, such as FreeBSD @command{/bin/sh} or @command{ash}, have -bugs when using @command{!} in compound commands: +bugs when using @command{!}: @example -$ @kbd{sh -c 'true && ! true | true; echo $?'} +$ @kbd{sh -c '! : | :'; echo $?} 1 -$ @kbd{ash -c 'true && ! true | true; echo $?'} +$ @kbd{ash -c '! : | :'; echo $?} 0 +$ @kbd{sh -c '! @{ :; @}'; echo $?} +1 +$ @kbd{ash -c '! @{ :; @}'; echo $?} +@{: not found +Syntax error: "@}" unexpected +2 @end example Shell code like this: -- 1.5.5.1 From MAILER-DAEMON Sun May 18 05:15:27 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JxezH-0003Uf-0w for mharc-autoconf-patches@gnu.org; Sun, 18 May 2008 05:15:27 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JxezB-0003Tu-8N for autoconf-patches@gnu.org; Sun, 18 May 2008 05:15:21 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Jxez9-0003TG-Dt for autoconf-patches@gnu.org; Sun, 18 May 2008 05:15:20 -0400 Received: from [199.232.76.173] (port=57300 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Jxez7-0003So-Bj for autoconf-patches@gnu.org; Sun, 18 May 2008 05:15:18 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:33641) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1Jxez7-0002da-0y for autoconf-patches@gnu.org; Sun, 18 May 2008 05:15:17 -0400 Received: from localhost.localdomain (xdsl-87-78-163-238.netcologne.de [87.78.163.238]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id 1D177400016EF; Sun, 18 May 2008 11:15:15 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1Jxeyu-0002PQ-65; Sun, 18 May 2008 11:15:04 +0200 Date: Sun, 18 May 2008 11:15:04 +0200 From: Ralf Wildenhues To: Eric Blake Message-ID: <20080518091503.GB9100@ins.uni-bonn.de> Mail-Followup-To: Eric Blake , autoconf-patches@gnu.org References: <20080506204153.GC16385@ins.uni-bonn.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Cc: autoconf-patches@gnu.org Subject: Re: let AC_MSG_FAILURE report pwd X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 May 2008 09:15:21 -0000 Hi Eric, * Eric Blake wrote on Mon, May 12, 2008 at 03:59:36PM CEST: > > > People simply don't realize that there could be more than one config.log > > > and config.cache file. > > > > > > > > OK to apply? > > > > Yes please. > > I took the liberty of applying it on your behalf. Thank you, and sorry that I didn't get to it. Cheers, Ralf From MAILER-DAEMON Sun May 18 12:06:15 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JxlOp-0008Sq-Ff for mharc-autoconf-patches@gnu.org; Sun, 18 May 2008 12:06:15 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JxlOo-0008RQ-C8 for autoconf-patches@gnu.org; Sun, 18 May 2008 12:06:14 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JxlOm-0008PM-Az for autoconf-patches@gnu.org; Sun, 18 May 2008 12:06:13 -0400 Received: from [199.232.76.173] (port=38526 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JxlOm-0008P4-1a for autoconf-patches@gnu.org; Sun, 18 May 2008 12:06:12 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:33467) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JxlOl-0005Rq-FL for autoconf-patches@gnu.org; Sun, 18 May 2008 12:06:11 -0400 Received: from localhost.localdomain (xdsl-87-78-90-69.netcologne.de [87.78.90.69]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id 8645040001D27; Sun, 18 May 2008 18:06:10 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1JxlOZ-0002n9-Bz; Sun, 18 May 2008 18:05:59 +0200 Date: Sun, 18 May 2008 18:05:59 +0200 From: Ralf Wildenhues To: Didier Barvaux Message-ID: <20080518160558.GA10720@ins.uni-bonn.de> Mail-Followup-To: Didier Barvaux , autoconf-patches@gnu.org References: <20080510173757.551c1bf9@shou.barvaux.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080510173757.551c1bf9@shou.barvaux.org> Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Cc: autoconf-patches@gnu.org Subject: Re: Extend AT_CHECK_MACRO to test macros with -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 May 2008 16:06:14 -0000 * Didier Barvaux wrote on Sat, May 10, 2008 at 05:37:57PM CEST: > I encountered a problem with the AC_C_CONST macro when using -O2 -Wall > -Werror as CFLAGS. The AC_C_CONST macro broke with similar CFLAGS some > time ago [1]. Thus I created a patch that extends the AT_CHECK_MACRO > macro used in the testsuite. > +# We run `configure' one more time with CFLAGS to check that unused or > +# uninitialized variables do not change the macro result. I understand the motivation behind this patch. But it should be done carefully, if at all possible. Different GCC versions emit different sets of warnings. Also, some other compilers may understand -O2 -Wall -Werror (Intel, PathScale?, Portland?) and many more do not. In general, it is really impossible to write Autoconf tests in a way such that no compiler emits warnings at all. Thus enabling -Werror unconditionally is somewhat of a bad idea. There is still the pending patch submission to have more flexible AC_LANG_WERROR. Unfortunately, my time to work on Autoconf is currently quite limited... Cheers, Ralf From MAILER-DAEMON Sun May 18 12:09:09 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JxlRd-0004wn-5i for mharc-autoconf-patches@gnu.org; Sun, 18 May 2008 12:09:09 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JxlRb-0004ux-Qf for autoconf-patches@gnu.org; Sun, 18 May 2008 12:09:07 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JxlRa-0004sy-77 for autoconf-patches@gnu.org; Sun, 18 May 2008 12:09:07 -0400 Received: from [199.232.76.173] (port=38679 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JxlRZ-0004sb-Of for autoconf-patches@gnu.org; Sun, 18 May 2008 12:09:05 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:33680) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JxlRY-0006Dr-OZ for autoconf-patches@gnu.org; Sun, 18 May 2008 12:09:04 -0400 Received: from localhost.localdomain (xdsl-87-78-90-69.netcologne.de [87.78.90.69]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id 17ACD40000E3A; Sun, 18 May 2008 18:09:04 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1JxlRN-0002nN-4Y; Sun, 18 May 2008 18:08:53 +0200 Date: Sun, 18 May 2008 18:08:53 +0200 From: Ralf Wildenhues To: Didier Barvaux , autoconf-patches@gnu.org Message-ID: <20080518160852.GB10720@ins.uni-bonn.de> Mail-Followup-To: Didier Barvaux , autoconf-patches@gnu.org References: <20080510173757.551c1bf9@shou.barvaux.org> <20080518160558.GA10720@ins.uni-bonn.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080518160558.GA10720@ins.uni-bonn.de> Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Cc: Subject: Re: Extend AT_CHECK_MACRO to test macros with -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 May 2008 16:09:08 -0000 * Ralf Wildenhues wrote on Sun, May 18, 2008 at 06:05:59PM CEST: > * Didier Barvaux wrote on Sat, May 10, 2008 at 05:37:57PM CEST: > > +# We run `configure' one more time with CFLAGS to check that unused or > > +# uninitialized variables do not change the macro result. > > I understand the motivation behind this patch. One more thing: how much of a time increase does this patch cause for a testsuite run? At a glance it could almost double it? Thanks, Ralf From MAILER-DAEMON Sun May 18 12:14:44 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1JxlX2-0002XF-8m for mharc-autoconf-patches@gnu.org; Sun, 18 May 2008 12:14:44 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JxlWz-0002Wb-Gb for autoconf-patches@gnu.org; Sun, 18 May 2008 12:14:41 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JxlWx-0002WL-VE for autoconf-patches@gnu.org; Sun, 18 May 2008 12:14:41 -0400 Received: from [199.232.76.173] (port=53533 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JxlWx-0002WI-PE for autoconf-patches@gnu.org; Sun, 18 May 2008 12:14:39 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:48079) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JxlWw-0007TZ-P9 for autoconf-patches@gnu.org; Sun, 18 May 2008 12:14:39 -0400 Received: from localhost.localdomain (xdsl-87-78-90-69.netcologne.de [87.78.90.69]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id DBE0C4000099A; Sun, 18 May 2008 18:14:37 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1JxlWk-0002nz-Jy; Sun, 18 May 2008 18:14:26 +0200 Date: Sun, 18 May 2008 18:14:26 +0200 From: Ralf Wildenhues To: Didier Barvaux Message-ID: <20080518161426.GC10720@ins.uni-bonn.de> Mail-Followup-To: Didier Barvaux , autoconf-patches@gnu.org References: <20080510173725.054b971a@shou.barvaux.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080510173725.054b971a@shou.barvaux.org> Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Cc: autoconf-patches@gnu.org Subject: Re: fix AC_C_CONST to work with CFLAGS -O2 -Wall -Werror X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 May 2008 16:14:41 -0000 * Didier Barvaux wrote on Sat, May 10, 2008 at 05:37:25PM CEST: > --- a/lib/autoconf/c.m4 > +++ b/lib/autoconf/c.m4 > @@ -1587,7 +1587,7 @@ AC_DEFUN([AC_C_CONST], > #ifndef __cplusplus > /* Ultrix mips cc rejects this. */ > typedef int charset[2]; > - const charset cs; > + const charset cs = { 0, 0 }; > /* SunOS 4.1.1 cc rejects this. */ > char const *const *pcpcc; > char **ppc; I'm pretty sure this kind of change has the potential to not any more expose the compiler bug this originally was done for. So really the solution would be to not use AC_C_CONST; Autoconf should warn about it as obsolete (and not only list it as obsolescent in NEWS) and autoscan should not recommend it any more. Cheers, Ralf From MAILER-DAEMON Mon May 26 01:48:00 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K0VYu-0000sX-FO for mharc-autoconf-patches@gnu.org; Mon, 26 May 2008 01:48:00 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K0VYs-0000sI-Co for autoconf-patches@gnu.org; Mon, 26 May 2008 01:47:58 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K0VYq-0000s6-KA for autoconf-patches@gnu.org; Mon, 26 May 2008 01:47:56 -0400 Received: from [199.232.76.173] (port=56918 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K0VYq-0000s3-HS for autoconf-patches@gnu.org; Mon, 26 May 2008 01:47:56 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:44435) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K0VYp-00071r-Ue for autoconf-patches@gnu.org; Mon, 26 May 2008 01:47:56 -0400 Received: from localhost.localdomain (xdsl-87-78-133-202.netcologne.de [87.78.133.202]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id 0F8DA40000E13 for ; Mon, 26 May 2008 07:47:55 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1K0VYd-000333-3M for autoconf-patches@gnu.org; Mon, 26 May 2008 07:47:43 +0200 Date: Mon, 26 May 2008 07:47:43 +0200 From: Ralf Wildenhues To: autoconf-patches@gnu.org Message-ID: <20080526054742.GA11702@ins.uni-bonn.de> Mail-Followup-To: autoconf-patches@gnu.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Subject: parallel autotest [0/3] X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2008 05:47:58 -0000 There are many possible ways to parallelize Autotest testsuite execution, for example: a) have each test group be a 'make' target in a makefile, then use parallel make. b) leverage the job server from GNU make, c) reimplement in shell a simplified job server a la parallel GNU make, d) implement a "worker thread" parallelization in shell. Features and differences of these approaches: - (a) and (b) need GNU make for parallel execution, - AFAICS (b) currently needs a shell that understands 'read -n1', - (d) differs from (c) in that one subprocess executes more than one test group, thus is potentially faster because it forks less, - (b) has the nice feature that it allows to parallelize across multiple test suites, and across testing and other, independent build activity. That means, while (c) and (d) allow make check TESTSUITEFLAGS='-j3' to speed up things, (b) allows gmake -j3 check to profit. I've experimented a bit with these approaches. I did not see an easy way to get (a) to work under the restrictions that it may not start the complete testsuite anew for each job: this has both very high overhead, and/or it requires that user-provided startup bits like atlocal be idempotent. I must confess that I didn't try very hard, though. This patch series consists of 1) a preparatory patch refactoring the driver loop, 2) an experimental patch implementing (c) if the system has mkfifo, 3) an *experimental* patch supporting (b) if GNU make is used, and the shell can 'read -n1'. The patches (2) and (3) currently both have - an unknown number of remaining race conditions, ;-) - known file descriptor leaks to test grous (not sure whether to view that as a problem or not), - the bug^Wlimitation that, when a parallel run is interrupted, the currently running test groups may still finish (see below also), - tests that are probably still too strict. Right now, the only system where I had significant problems was Cygwin with its seemingly limited named fifo emulation. I expect that, given sufficient interest, somebody will fix this for me. ;-) (Of course, (c) doesn't work with MinGW, as it has no named fifos.) Also, I am reluctant to apply (3) before asking for permission on the GNU make mailing list. I guess I should comment on (d): Tried out an implementation where an empty directory is created for each test group. Each worker thread tries to rmdir the next available dir, and if it succeeds, runs that test group, and writes out the results in a (different) per-test-group directory. This works, doesn't cause too much "lock contention" on practical loads, but could do so in theory. However, it is a bit tough on the file system, 'make check' on tmpfs being noticeably faster than on NFS. (Probably the latter holds for the upcoming patches too, hopefully not as bad though; I haven't measured). I haven't tried this alternative implementation for (d) yet: When ready for another test group, a worker thread p writes one byte which encodes $p, to pipe-M, then waits reading the number of the next test group from the per-worker pipe-$p. This limits parallelism to 250odd workers, as pipes do not provide for atomic transfer of multiple bytes. I even wonder whether we may count on data not being dropped with multiple writers to a pipe (this may be a reason for the Cygwin issues mentioned above). A comment on interrupting execution: My incomplete testing and exploration of interrupting the testsuite has not shown up a decent and portable way yet. Some shells seem to behave sanely, with others I am wondering whether I have merely assumed too much bash-like semantics, or it's really impossible to do right. Non-unix systems are a completely different story, too. Anyway it would be good to have this fixed before parallel autotest is a non-experimental feature. Comments, reports etc. appreciated. The GNU make job server is described in . Cheers, Ralf From MAILER-DAEMON Mon May 26 01:50:22 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K0VbC-0001OV-8b for mharc-autoconf-patches@gnu.org; Mon, 26 May 2008 01:50:22 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K0VbA-0001MT-Fq for autoconf-patches@gnu.org; Mon, 26 May 2008 01:50:20 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K0Vb9-0001Lg-PU for autoconf-patches@gnu.org; Mon, 26 May 2008 01:50:20 -0400 Received: from [199.232.76.173] (port=49560 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K0Vb9-0001LW-70 for autoconf-patches@gnu.org; Mon, 26 May 2008 01:50:19 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:44557) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K0Vb8-0007ov-ET for autoconf-patches@gnu.org; Mon, 26 May 2008 01:50:18 -0400 Received: from localhost.localdomain (xdsl-87-78-133-202.netcologne.de [87.78.133.202]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id 2F6E1400009FC for ; Mon, 26 May 2008 07:50:17 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1K0Vav-00033o-4M for autoconf-patches@gnu.org; Mon, 26 May 2008 07:50:05 +0200 Date: Mon, 26 May 2008 07:50:05 +0200 From: Ralf Wildenhues To: autoconf-patches@gnu.org Message-ID: <20080526055004.GB11702@ins.uni-bonn.de> Mail-Followup-To: autoconf-patches@gnu.org References: <20080526054742.GA11702@ins.uni-bonn.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20080526054742.GA11702@ins.uni-bonn.de> Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Subject: parallel autotest [1/3]: Refactor testsuite driver loop. X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2008 05:50:20 -0000 2008-04-13 Ralf Wildenhues * lib/autotest/general.m4 (AS_MESSAGE_LOG_FD): Move definition earlier in the file. (AT_INIT): Create line number cache in $at_suite_dir/at-source-lines. : New directory at-groups below $at_suite_dir. (at_func_group_prepare, at_func_group_postprocess): New shell functions to factorize per-test group work. Keep the actual test execution outside of a shell function in order to avoid zsh 4.x exit status bugs. : Turn these into per-group files below $at_helper_dir. Also store test results there in files named pass, fail, xpass, xfail, skip. Let the parent collect results from $at_helper_dir. Adjust summary statistics computation. diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4 index 4ce3df7..cfa18fc 100644 --- a/lib/autotest/general.m4 +++ b/lib/autotest/general.m4 @@ -215,6 +215,8 @@ m4_define([AT_groups_all], []) m4_define([AT_help_all], []) m4_foreach([AT_name], [_AT_DEFINE_INIT_LIST], [m4_popdef(m4_defn([AT_name]= ))]) m4_wrap([_AT_FINISH]) +dnl Define FDs. +m4_define([AS_MESSAGE_LOG_FD], [5]) AS_INIT[]dnl m4_divert_push([DEFAULTS])dnl AT_COPYRIGHT( @@ -766,18 +768,10 @@ fi at_suite_dir=3D$at_dir/$as_me.dir # The file containing the suite. at_suite_log=3D$at_dir/$as_me.log -# The file containing the location of the last AT_CHECK. -at_check_line_file=3D$at_suite_dir/at-check-line -# The file containing the exit status of the last command. -at_status_file=3D$at_suite_dir/at-status -# The files containing the output of the tested commands. -at_stdout=3D$at_suite_dir/at-stdout -at_stder1=3D$at_suite_dir/at-stder1 -at_stderr=3D$at_suite_dir/at-stderr -# The file containing the function to run a test group. -at_test_source=3D$at_suite_dir/at-test-source -# The file containing dates. -at_times_file=3D$at_suite_dir/at-times +# The directory containing helper files per test group. +at_helper_dir=3D$at_suite_dir/at-groups +# Stop file: if it exists, do not start new jobs. +at_stop_file=3D$at_suite_dir/at-stop =20 if $at_clean; then test -d "$at_suite_dir" && @@ -835,7 +829,7 @@ export PATH =20 # Setting up the FDs. # 5 is the log file. Not to be overwritten if `-d'. -m4_define([AS_MESSAGE_LOG_FD], [5]) +dnl FDs are defined earlier in this file. if $at_debug_p; then at_suite_log=3D/dev/null else @@ -915,12 +909,6 @@ done at_start_date=3D`date` at_start_time=3D`date +%s 2>/dev/null` AS_ECHO(["$as_me: starting at: $at_start_date"]) >&AS_MESSAGE_LOG_FD -at_xpass_list=3D -at_xfail_list=3D -at_pass_list=3D -at_fail_list=3D -at_skip_list=3D -at_group_count=3D0 m4_divert_pop([PREPARE_TESTS])dnl m4_divert_push([TESTS])dnl =20 @@ -959,19 +947,47 @@ BEGIN { FS=3D"=07" } test =3D substr ($ 0, 10) print "at_sed" test "=3D\"1," start "d;" (NR-1) "q\"" if (test =3D=3D "'"$at_group"'") exit -}' "$at_myself" > "$at_test_source" && -. "$at_test_source" || +}' "$at_myself" > "$at_suite_dir/at-source-lines" && +. "$at_suite_dir/at-source-lines" || AS_ERROR([cannot create test line number cache]) +rm -f "$at_suite_dir/at-source-lines" =20 +# Set up helper dirs. +rm -rf "$at_helper_dir" && +mkdir "$at_helper_dir" && +cd "$at_helper_dir" && +{ test -z "$at_groups" || mkdir $at_groups; } || +AS_ERROR([testsuite directory setup failed]) + +# Functions for running a test group. We leave the actual +# test group execution outside of a shell function in order +# to avoid hitting zsh 4.x exit status bugs. + +# at_func_group_prepare +# --------------------- +# Prepare running a test group +at_func_group_prepare () +{ + # The directory for additional per-group helper files. + at_job_dir=3D$at_helper_dir/$at_group + # The file containing the location of the last AT_CHECK. + at_check_line_file=3D$at_job_dir/at-check-line + # The file containing the exit status of the last command. + at_status_file=3D$at_job_dir/at-status + # The files containing the output of the tested commands. + at_stdout=3D$at_job_dir/at-stdout + at_stder1=3D$at_job_dir/at-stder1 + at_stderr=3D$at_job_dir/at-stderr + # The file containing the code for a test group. + at_test_source=3D$at_job_dir/at-test-source + # The file containing dates. + at_times_file=3D$at_job_dir/at-times =20 -m4_text_box([Driver loop.]) -for at_group in $at_groups -do # Be sure to come back to the top test directory. cd "$at_suite_dir" =20 # Clearly separate the test groups when verbose. - test $at_group_count !=3D 0 && $at_verbose echo + $at_first || $at_verbose echo =20 at_group_normalized=3D$at_group _AT_NORMALIZE_TEST_GROUP_NUMBER(at_group_normalized) @@ -982,70 +998,68 @@ do if test -d "$at_group_dir"; then find "$at_group_dir" -type d ! -perm -700 -exec chmod u+rwx \{\} \; rm -fr "$at_group_dir" || - AS_WARN([test directory could not be cleaned.]) + AS_WARN([test directory for $at_group_normalized could not be cleaned.= ]) fi # Be tolerant if the above `rm' was not able to remove the directory. AS_MKDIR_P(["$at_group_dir"]) - cd "$at_group_dir" =20 echo 0 > "$at_status_file" =20 # In verbose mode, append to the log file *and* show on - # the standard output; in quiet mode only write to the log + # the standard output; in quiet mode only write to the log. if test -z "$at_verbose"; then at_tee_pipe=3D'tee -a "$at_group_log"' else at_tee_pipe=3D'cat >> "$at_group_log"' fi +} =20 - if at_func_test $at_group && . "$at_test_source"; then :; else - AS_ECHO(["$as_me: unable to parse test group: $at_group"]) >&2 - at_failed=3D: - fi - +# at_func_group_postprocess +# ------------------------- +at_func_group_postprocess () +{ # Be sure to come back to the suite directory, in particular # since below we might `rm' the group directory we are in currently. cd "$at_suite_dir" =20 if test ! -f "$at_check_line_file"; then - sed "s/^ */$as_me: warning: /" <<_ATEOF - A failure happened in a test group before any test could be - run. This means that test suite is improperly designed. Please - report this failure to . + sed "s/^ */$as_me: WARNING: /" <<_ATEOF + A failure happened in a test group before any test could be + run. This means that test suite is improperly designed. Please + report this failure to . _ATEOF AS_ECHO(["$at_setup_line"]) >"$at_check_line_file" fi - at_func_arith 1 + $at_group_count - at_group_count=3D$at_func_arith_result $at_verbose AS_ECHO_N(["$at_group. $at_setup_line: "]) AS_ECHO_N(["$at_group. $at_setup_line: "]) >> "$at_group_log" case $at_xfail:$at_status in yes:0) at_msg=3D"UNEXPECTED PASS" - at_xpass_list=3D"$at_xpass_list $at_group" + at_res=3Dxpass at_errexit=3D$at_errexit_p ;; no:0) at_msg=3D"ok" - at_pass_list=3D"$at_pass_list $at_group" + at_res=3Dpass at_errexit=3Dfalse ;; *:77) at_msg=3D'skipped ('`cat "$at_check_line_file"`')' - at_skip_list=3D"$at_skip_list $at_group" + at_res=3Dskip at_errexit=3Dfalse ;; yes:*) at_msg=3D'expected failure ('`cat "$at_check_line_file"`')' - at_xfail_list=3D"$at_xfail_list $at_group" + at_res=3Dxfail at_errexit=3Dfalse ;; no:*) at_msg=3D'FAILED ('`cat "$at_check_line_file"`')' - at_fail_list=3D"$at_fail_list $at_group" + at_res=3Dfail at_errexit=3D$at_errexit_p ;; esac + echo "$at_res" > "$at_job_dir/$at_res" # Make sure there is a separator even with long titles. AS_ECHO([" $at_msg"]) at_log_msg=3D"$at_group. $at_desc ($at_setup_line): $at_msg" @@ -1063,7 +1077,7 @@ _ATEOF AS_ECHO(["$at_log_msg"]) >&AS_MESSAGE_LOG_FD =20 # Cleanup the group directory, unless the user wants the files. - if $at_debug_p ; then + if $at_debug_p; then at_func_create_debugging_script else if test -d "$at_group_dir"; then @@ -1079,16 +1093,61 @@ _ATEOF # is later included in the global log. AS_ECHO(["$at_log_msg"]) >> "$at_group_log" =20 - # Upon failure, keep the group directory for autopsy, and - # create the debugging script. + # Upon failure, keep the group directory for autopsy, and create + # the debugging script. With -e, do not start any further tests. at_func_create_debugging_script - $at_errexit && break + if $at_errexit; then + echo stop > "$at_stop_file" + fi ;; esac +} + + +m4_text_box([Driver loop.]) + +rm -f "$at_stop_file" +at_first=3D: + +for at_group in $at_groups; do + at_func_group_prepare + if cd "$at_group_dir" && + at_func_test $at_group && + . "$at_test_source"; then :; else + AS_WARN([unable to parse test group: $at_group]) + at_failed=3D: + fi + at_func_group_postprocess + test -f "$at_stop_file" && break + at_first=3Dfalse done =20 +# Wrap up the test suite with summary statistics. +cd "$at_helper_dir" + +at_pass_list=3D`echo */pass | sed 's,/pass,,g; s,\*,,'` +at_xpass_list=3D`echo */xpass | sed 's,/xpass,,g; s,\*,,'` +at_xfail_list=3D`echo */xfail | sed 's,/xfail,,g; s,\*,,'` +at_fail_list=3D`echo */fail | sed 's,/fail,,g; s,\*,,'` +at_skip_list=3D`echo */skip | sed 's,/skip,,g; s,\*,,'` + +set X $at_pass_list $at_xpass_list $at_xfail_list $at_fail_list $at_skip_l= ist +shift; at_group_count=3D$[@%:@] +set X $at_xpass_list; shift; at_xpass_count=3D$[@%:@] +set X $at_xfail_list; shift; at_xfail_count=3D$[@%:@] +set X $at_fail_list; shift; at_fail_count=3D$[@%:@] +set X $at_skip_list; shift; at_skip_count=3D$[@%:@] + +at_func_arith $at_group_count - $at_skip_count +at_run_count=3D$at_func_arith_result +at_func_arith $at_xpass_count + $at_fail_count +at_unexpected_count=3D$at_func_arith_result +at_func_arith $at_xfail_count + $at_fail_count +at_total_fail_count=3D$at_func_arith_result + # Back to the top directory. cd "$at_dir" +rm -rf "$at_helper_dir" =20 # Compute the duration of the suite. at_stop_date=3D`date` @@ -1111,19 +1170,6 @@ case $at_start_time,$at_stop_time in ;; esac =20 -# Wrap up the test suite with summary statistics. -set X $at_skip_list; shift; at_skip_count=3D$[@%:@] -set X $at_fail_list; shift; at_fail_count=3D$[@%:@] -set X $at_xpass_list; shift; at_xpass_count=3D$[@%:@] -set X $at_xfail_list; shift; at_xfail_count=3D$[@%:@] - -at_func_arith $at_group_count - $at_skip_count -at_run_count=3D$at_func_arith_result -at_func_arith $at_xpass_count + $at_fail_count -at_unexpected_count=3D$at_func_arith_result -at_func_arith $at_xfail_count + $at_fail_count -at_total_fail_count=3D$at_func_arith_result - echo AS_BOX([Test results.]) echo From MAILER-DAEMON Mon May 26 01:51:11 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K0Vbz-00024r-Co for mharc-autoconf-patches@gnu.org; Mon, 26 May 2008 01:51:11 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K0Vbw-00024L-KS for autoconf-patches@gnu.org; Mon, 26 May 2008 01:51:08 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K0Vbt-000240-6C for autoconf-patches@gnu.org; Mon, 26 May 2008 01:51:07 -0400 Received: from [199.232.76.173] (port=49571 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K0Vbs-00023i-Se for autoconf-patches@gnu.org; Mon, 26 May 2008 01:51:04 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:39584) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K0Vbr-00081a-Vd for autoconf-patches@gnu.org; Mon, 26 May 2008 01:51:04 -0400 Received: from localhost.localdomain (xdsl-87-78-133-202.netcologne.de [87.78.133.202]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id DB99940000BE1 for ; Mon, 26 May 2008 07:51:02 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1K0Vbe-000342-Ub for autoconf-patches@gnu.org; Mon, 26 May 2008 07:50:50 +0200 Date: Mon, 26 May 2008 07:50:50 +0200 From: Ralf Wildenhues To: autoconf-patches@gnu.org Message-ID: <20080526055050.GC11702@ins.uni-bonn.de> Mail-Followup-To: autoconf-patches@gnu.org References: <20080526054742.GA11702@ins.uni-bonn.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20080526054742.GA11702@ins.uni-bonn.de> Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Subject: parallel autotest [2/3]: Implement 'testsuite --jobs'. X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2008 05:51:08 -0000 There may be systems that support mknod but not mkfifo. Not sure whether it's useful to support them. Cheers, Ralf 2008-05-26 Ralf Wildenhues Implement parallel Autotest test execution: testsuite --jobs. * lib/autotest/general.m4 (AT_JOB_FIFO_FD): New macro. (AT_INIT): : New variable. Accept -j, -jN, --jobs[=3DN], document them in --help output. Implement parallel driver loop using a FIFO, enabled with --jobs and if mkfifo works; otherwise, fall back to sequential loop. (AT_SETUP): Store, do not output summary progress line if parallel. * tests/autotest.at (parallel test execution, parallel truth) (parallel fallacy, parallel skip): New tests. * doc/autoconf.texi (testsuite Invocation): Document -j, --jobs, the mkfifo requirement, and that --errexit may cause concurrent jobs to finish. * NEWS: Update. diff --git a/NEWS b/NEWS index 9396ac0..fdee3e0 100644 --- a/NEWS +++ b/NEWS @@ -2,6 +2,8 @@ GNU Autoconf NEWS - User visible changes. =20 * Major changes in Autoconf 2.62a (2008-??-??) =20 +** Autotest testsuites accept an option --jobs[=3DN] for parallel testing. + =0C * Major changes in Autoconf 2.62 (2008-04-05) [stable] Released by Eric Blake, based on git versions 2.61a.*. diff --git a/doc/autoconf.texi b/doc/autoconf.texi index c22b822..82ee582 100644 --- a/doc/autoconf.texi +++ b/doc/autoconf.texi @@ -20325,6 +20325,24 @@ Change the current directory to @var{dir} before c= reating any files. Useful for running the testsuite in a subdirectory from a top-level Makefile. =20 +@item --jobs@r{[}=3D@var{n}@r{]} +@itemx -j@ovar{n} +@itemx -j +Run @var{n} tests in parallel, if possible. If @var{n} is not given, +run all given tests in parallel. Note that there should be no space +before the argument to @option{-j}, as @option{-j @var{number}} denotes +the separate arguments @option{-j} and @option{@var{number}}, see below. + +In parallel mode, the standard input device of the testsuite script is +not available to commands inside a test group. Furthermore, banner +lines are not printed, and the summary line for each test group is +output after the test group completes. Summary lines may appear +unordered. If verbose and trace output are enabled (see below), they +may appear intermixed from concurrently running tests. + +Parallel mode requires the @command{mkfifo} command to work, and will be +silently disabled otherwise. + @item --clean @itemx -c Remove all the files the test suite might have created and exit. Meant @@ -20402,6 +20420,8 @@ If any test fails, immediately abort testing. It i= mplies @option{--debug}: post test group clean up, and top-level logging are inhibited. This option is meant for the full test suite, it is not really useful for generated debugging scripts. +If the testsuite is run in parallel mode using @option{--jobs}, +then concurrently running tests will finish before exiting. =20 @item --verbose @itemx -v diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4 index cfa18fc..f620561 100644 --- a/lib/autotest/general.m4 +++ b/lib/autotest/general.m4 @@ -217,6 +217,7 @@ m4_foreach([AT_name], [_AT_DEFINE_INIT_LIST], [m4_popde= f(m4_defn([AT_name]))]) m4_wrap([_AT_FINISH]) dnl Define FDs. m4_define([AS_MESSAGE_LOG_FD], [5]) +m4_define([AT_JOB_FIFO_FD], [6]) AS_INIT[]dnl m4_divert_push([DEFAULTS])dnl AT_COPYRIGHT( @@ -398,6 +399,8 @@ at_errexit_p=3Dfalse # Shall we be verbose? ':' means no, empty means yes. at_verbose=3D: at_quiet=3D +# Running several jobs in parallel, 0 means as many as test groups. +at_jobs=3D1 =20 # Shall we keep the debug scripts? Must be `:' when the suite is # run by a debug script, so that the script doesn't remove itself. @@ -558,6 +561,21 @@ do at_dir=3D$at_optarg ;; =20 + # Parallel execution. + --jobs | -j ) + at_jobs=3D0 + ;; + --jobs=3D* | -j[[0-9]]* ) + if test -n "$at_optarg"; then + at_jobs=3D$at_optarg + else + at_jobs=3D`expr X$at_option : 'X-j\(.*\)'` + fi + case $at_jobs in *[[!0-9]]*) + AS_ERROR([non-numeric argument to -j/--jobs: $at_jobs]) ;; + esac + ;; + # Keywords. --keywords | -k ) at_prev=3D--keywords @@ -662,6 +680,8 @@ dnl extra quoting prevents emacs whitespace mode from p= utting tabs in output Execution tuning: -C, --directory=3DDIR [ change to directory DIR before starting] + -j[[N]], --jobs[[=3DN]] +[ Allow N jobs at once; infinite jobs with no arg] -k, --keywords=3DKEYWORDS [ select the tests matching all the comma-separated KEYWOR= DS] [ multiple \`-k' accumulate; prefixed \`!' negates a KEYWO= RD] @@ -772,6 +792,8 @@ at_suite_log=3D$at_dir/$as_me.log at_helper_dir=3D$at_suite_dir/at-groups # Stop file: if it exists, do not start new jobs. at_stop_file=3D$at_suite_dir/at-stop +# The fifo used for the job dispatcher. +at_job_fifo=3D$at_suite_dir/at-job-fifo =20 if $at_clean; then test -d "$at_suite_dir" && @@ -952,6 +974,12 @@ BEGIN { FS=3D"=07" } AS_ERROR([cannot create test line number cache]) rm -f "$at_suite_dir/at-source-lines" =20 +# If parallel mode, don't output banners, don't split summary lines. +if test $at_jobs -ne 1; then + at_print_banners=3Dfalse + at_quiet=3D: +fi + # Set up helper dirs. rm -rf "$at_helper_dir" && mkdir "$at_helper_dir" && @@ -1060,8 +1088,13 @@ _ATEOF ;; esac echo "$at_res" > "$at_job_dir/$at_res" - # Make sure there is a separator even with long titles. - AS_ECHO([" $at_msg"]) + # In parallel mode, output the summary line only afterwards. + if test $at_jobs -ne 1 && test -n "$at_verbose"; then + AS_ECHO(["$at_desc_line $at_msg"]) + else + # Make sure there is a separator even with long titles. + AS_ECHO([" $at_msg"]) + fi at_log_msg=3D"$at_group. $at_desc ($at_setup_line): $at_msg" case $at_status in 0|77) @@ -1107,20 +1140,77 @@ _ATEOF m4_text_box([Driver loop.]) =20 rm -f "$at_stop_file" +trap 'exit_status=3D$? + echo "signal received, bailing out" >&2 + echo stop > "$at_stop_file" + exit $exit_status' 1 2 13 15 at_first=3D: =20 -for at_group in $at_groups; do - at_func_group_prepare - if cd "$at_group_dir" && - at_func_test $at_group && - . "$at_test_source"; then :; else - AS_WARN([unable to parse test group: $at_group]) - at_failed=3D: +if test $at_jobs -ne 1 && + rm -f "$at_job_fifo" && + ( mkfifo "$at_job_fifo" ) 2>/dev/null && + exec AT_JOB_FIFO_FD<> "$at_job_fifo" +then + # FIFO job dispatcher. + echo + if test $at_jobs -eq 0; then + set X $at_groups; shift; at_jobs=3D$[@%:@] fi - at_func_group_postprocess - test -f "$at_stop_file" && break - at_first=3Dfalse -done + # Turn jobs into a list of numbers, starting from 1. + at_joblist=3D`AS_ECHO([" $at_groups_all "]) | \ + sed -e 's/\( '$at_jobs'\) .*/\1/'` + + set X $at_joblist + shift + for at_group in $at_groups; do + ( + # Start one test group. + at_func_group_prepare + if cd "$at_group_dir" && + at_func_test $at_group && + . "$at_test_source" # AT_JOB_FIFO_FD<&- + then :; else + AS_WARN([unable to parse test group: $at_group]) + at_failed=3D: + fi + at_func_group_postprocess + echo token >&AT_JOB_FIFO_FD + ) & + shift # Consume one token. + if test $[@%:@] -gt 0; then :; else + read at_token <&AT_JOB_FIFO_FD || break + set x $[*] + fi + test -f "$at_stop_file" && break + at_first=3Dfalse + done + # Read back the remaining ($at_jobs - 1) tokens. + set X $at_joblist + shift + if test $[@%:@] -gt 0; then + shift + for at_job + do + read at_token + done <&AT_JOB_FIFO_FD + fi + exec AT_JOB_FIFO_FD<&- + wait +else + # Run serially, avoid forks and other potential surprises. + for at_group in $at_groups; do + at_func_group_prepare + if cd "$at_group_dir" && + at_func_test $at_group && + . "$at_test_source"; then :; else + AS_WARN([unable to parse test group: $at_group]) + at_failed=3D: + fi + at_func_group_postprocess + test -f "$at_stop_file" && break + at_first=3Dfalse + done +fi =20 # Wrap up the test suite with summary statistics. cd "$at_helper_dir" @@ -1486,8 +1576,9 @@ at_setup_line=3D'm4_defn([AT_line])' m4_if(AT_banner_ordinal, [0], [], [at_func_banner AT_banner_ordinal ])dnl at_desc=3D"AS_ESCAPE(m4_dquote(m4_defn([AT_description])))" -$at_quiet AS_ECHO_N([m4_format(["%3d: $at_desc%*s"], AT_ordinal, - m4_max(0, m4_eval(47 - m4_qlen(m4_defn([AT_description])))), [])]) +at_desc_line=3Dm4_format(["%3d: $at_desc%*s"], AT_ordinal, + m4_max(0, m4_eval(47 - m4_qlen(m4_defn([AT_description])))), []) +$at_quiet AS_ECHO_N(["$at_desc_line"]) m4_divert_push([TEST_SCRIPT])dnl ]) =20 diff --git a/tests/autotest.at b/tests/autotest.at index e6dc862..54399f3 100644 --- a/tests/autotest.at +++ b/tests/autotest.at @@ -745,6 +745,95 @@ AT_CHECK_KEYS([--list -k none -k first], [none|first],= [2], [second|both], [0]) AT_CLEANUP =20 =20 +## ----------------------- ## +## parallel test execution ## +## ----------------------- ## + +AT_SETUP([parallel test execution]) + +# The total number of tests for the parallel test micro-suite, +# the number of tests to run concurrently. +# +# The total number should not be too high, to not slow down +# the testsuite unnecessarily. If the number of concurrent +# jobs is too low, the race is lost more easily (see below), +# if it is too high, fork may fail on tightly limited systems. +m4_define([AT_PARALLEL_TESTS_TOTAL], [8]) +m4_define([AT_PARALLEL_TESTS_RUN], [4]) + + +AT_CHECK_AT_PREP([micro-suite], +[[AT_INIT([suite to test parallel execution]) +m4_for([count], [1], ]]AT_PARALLEL_TESTS_TOTAL[[, [], + [AT_SETUP([test number count]) + AT_CHECK([sleep 1]) + AT_CLEANUP +]) +]]) + +AT_CHECK([$CONFIG_SHELL ./micro-suite --help | grep " --jobs"], [0], [igno= re]) +AT_CHECK([$CONFIG_SHELL ./micro-suite -j2foo], [1], [], [stderr]) +AT_CHECK([grep 'non-numeric argument' stderr], [], [ignore]) +AT_CHECK([$CONFIG_SHELL ./micro-suite --jobs=3Dfoo], [1], [], [stderr]) +AT_CHECK([grep 'non-numeric argument' stderr], [], [ignore]) +AT_CHECK([$CONFIG_SHELL ./micro-suite -j[]AT_PARALLEL_TESTS_RUN], [], [std= out]) +# Ensure that all tests run, and lines are not split. +AT_CHECK([grep -c '^.\{53\}ok' stdout], [], [AT_PARALLEL_TESTS_TOTAL +]) + +# Ensure we really are faster than sequential execution: +# the testsuite should have completed before we kill it. +# Unfortunately, the return value of wait is unreliable, +# so we check that kill fails. +# +# Since the testsuite startup time can be high, we compare +# it against a sequential testsuite run in a subdirectory, +# where only a subset of tests is run. + +# The parallel scheduler requires mkfifo to work. +AT_CHECK([mkfifo fifo || exit 77]) +mkdir serial +AT_CHECK([$CONFIG_SHELL ./micro-suite --jobs=3D[]AT_PARALLEL_TESTS_RUN & ]= dnl + [sleep 3 && ]dnl + [cd serial && $CONFIG_SHELL ../micro-suite -3 >/dev/null && ]dnl + [{ kill $! && exit 1; :; }], [], [stdout], [ignore]) +AT_CHECK([grep -c '^.\{53\}ok' stdout], [], [AT_PARALLEL_TESTS_TOTAL +]) +AT_CHECK([grep 'AT_PARALLEL_TESTS_TOTAL tests' stdout], [], [ignore]) + +AT_CLEANUP + +AT_CHECK_AT_TEST([parallel truth], + [AT_CHECK([:], 0, [], [])], + [], [], [], [], + [], [-j]) + +AT_CHECK_AT_TEST([parallel fallacy], + [AT_CHECK([false], [], [], [])], + [], [1], [], [ignore], + [AT_CHECK([grep failed micro-suite.log], [], [ignore])], [-j]) + +AT_CHECK_AT_TEST([parallel skip], + [AT_CHECK([exit 77], 0, [], [])], + [], [], [], [], + [AT_CHECK([grep skipped micro-suite.log], [], [ignore])], [-j]) + +AT_CHECK_AT_TEST([parallel errexit], + [AT_CHECK([false]) + AT_CLEANUP + AT_SETUP([barrier test]) + AT_CHECK([sleep 4]) + AT_CLEANUP + AT_SETUP([test that should not be run]) + AT_CHECK([:])], + [], [1], [stdout], [stderr], + [AT_CHECK([test -f micro-suite.log], [1]) + touch micro-suite.log # shut up AT_CAPTURE_FILE. + AT_CHECK([grep "should not be run" stdout], [1]) + AT_CHECK([grep "[[12]] .* inhibited subsequent" stderr], [], [ignore])], + [-j2 --errexit]) + + ## ------------------- ## ## srcdir propagation. ## ## ------------------- ## From MAILER-DAEMON Mon May 26 01:53:39 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K0VeM-00034n-V8 for mharc-autoconf-patches@gnu.org; Mon, 26 May 2008 01:53:38 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K0VeL-00034i-L9 for autoconf-patches@gnu.org; Mon, 26 May 2008 01:53:37 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K0VeK-00034W-Te for autoconf-patches@gnu.org; Mon, 26 May 2008 01:53:37 -0400 Received: from [199.232.76.173] (port=54345 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K0VeK-00034T-R6 for autoconf-patches@gnu.org; Mon, 26 May 2008 01:53:36 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:39842) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K0VeK-00009Z-2E for autoconf-patches@gnu.org; Mon, 26 May 2008 01:53:36 -0400 Received: from localhost.localdomain (xdsl-87-78-133-202.netcologne.de [87.78.133.202]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id D019F400002D4 for ; Mon, 26 May 2008 07:53:34 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1K0Ve6-00034E-Sq for autoconf-patches@gnu.org; Mon, 26 May 2008 07:53:22 +0200 Date: Mon, 26 May 2008 07:53:22 +0200 From: Ralf Wildenhues To: autoconf-patches@gnu.org Message-ID: <20080526055322.GD11702@ins.uni-bonn.de> Mail-Followup-To: autoconf-patches@gnu.org References: <20080526054742.GA11702@ins.uni-bonn.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20080526054742.GA11702@ins.uni-bonn.de> Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Subject: parallel autotest [3/3]: GNU make jobserver client. X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2008 05:53:37 -0000 This patch, if to be used at all, would probably need something like AC_SUBST([at_parse_makeflags], ['at_dry_run=3D; at_flags=3D; at_first=3D:; \ for at_flag in $(MAKEFLAGS) --; do \ case $$at_flag in \ --jobserver-fds=3D*) at_flags=3D"$$at_flags $$at_flag";; \ -- | *=3D*) break;; \ --*) ;; \ *) \ if $$at_first; then \ case $$at_flag in *n*) at_dry_run=3D:;; esac; \ fi;; \ esac; \ at_first=3Dfalse; \ done']) to be generated by Autotest, for easier usage and to avoid duplication (also, documentation for it). Cheers, Ralf 2008-05-26 Ralf Wildenhues Implement experimental GNU make jobserver client support. * doc/autoconf.texi (testsuite Invocation): Document --jobserver-fds. * lib/autotest/general.m4 (AT_INIT): Accept --jobserver-fds, overriding --jobs if needed; document it in --help output. Move file descriptors in case of a clash with Autoconf's hard-coded ones. Enable support only if the shell is capable of one-character reads. Implement jobserver client for spawning test groups. * tests/Makefile.am (parse_makeflags): New macro. (check-local, installcheck-local): Use it to pass --jobserver-fds to the testsuite. * tests/autotest.at (parallel jobserver support): New test. diff --git a/NEWS b/NEWS index fdee3e0..be1f102 100644 --- a/NEWS +++ b/NEWS @@ -3,6 +3,7 @@ GNU Autoconf NEWS - User visible changes. * Major changes in Autoconf 2.62a (2008-??-??) =20 ** Autotest testsuites accept an option --jobs[=3DN] for parallel testing. + Alternatively, testsuites can act as GNU make jobserver client. =20 =0C * Major changes in Autoconf 2.62 (2008-04-05) [stable] diff --git a/doc/autoconf.texi b/doc/autoconf.texi index 82ee582..047c512 100644 --- a/doc/autoconf.texi +++ b/doc/autoconf.texi @@ -20343,6 +20343,12 @@ may appear intermixed from concurrently running te= sts. Parallel mode requires the @command{mkfifo} command to work, and will be silently disabled otherwise. =20 +@item --jobserver-fds=3D@var{m},@var{n} +Enable experimental parallel mode governed by a @acronym{GNU} @command{mak= e} +jobserver. This option should never be specified manually, but instead +be extracted from @code{$(MAKEFLAGS)} in the @file{Makefile} starting +the testsuite. It overrides a given @option{--jobs} option. + @item --clean @itemx -c Remove all the files the test suite might have created and exit. Meant diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4 index f620561..bd20617 100644 --- a/lib/autotest/general.m4 +++ b/lib/autotest/general.m4 @@ -401,6 +401,9 @@ at_verbose=3D: at_quiet=3D # Running several jobs in parallel, 0 means as many as test groups. at_jobs=3D1 +# File descriptors for the GNU make jobserver. +at_fd_in=3D +at_fd_out=3D =20 # Shall we keep the debug scripts? Must be `:' when the suite is # run by a debug script, so that the script doesn't remove itself. @@ -575,6 +578,37 @@ do AS_ERROR([non-numeric argument to -j/--jobs: $at_jobs]) ;; esac ;; + --jobserver-fds=3D* ) + at_save_IFS=3D$IFS + IFS=3D, + set X $at_optarg + IFS=3D$at_save_IFS + at_fd_in=3D$[2] + at_fd_out=3D$[3] + case $at_fd_in$at_fd_out in *[[!0-9]]*) + AS_ERROR([bogus argument $at_optarg]) ;; + esac + # Resolve file descriptor clashes. + if test $at_fd_in -eq AS_MESSAGE_LOG_FD; then + at_old_fd=3D$at_fd_in + at_fd_in=3Dm4_eval(AS_MESSAGE_LOG_FD + 1) + if test $at_fd_in -eq $at_fd_out; then + at_fd_in=3D`expr $at_fd_out + 1` + fi + eval exec "$at_fd_in<&$at_old_fd $at_old_fd<&-" + fi + if test $at_fd_out -eq AS_MESSAGE_LOG_FD; then + at_old_fd=3D$at_fd_out + at_fd_out=3Dm4_eval(AS_MESSAGE_LOG_FD + 2) + eval exec "$at_fd_out>&$at_old_fd $at_old_fd>&-" + fi + # Don't bother if the shell is not capable. + if eval '( AS_ECHO_N([++]) | read -n1 at_token ) 2>/dev/null'; then :; + else + eval "$at_fd_in<&- $at_fd_out>&-" + at_fd_in=3D; at_fd_out=3D + fi + ;; =20 # Keywords. --keywords | -k ) @@ -682,6 +716,8 @@ Execution tuning: [ change to directory DIR before starting] -j[[N]], --jobs[[=3DN]] [ Allow N jobs at once; infinite jobs with no arg] + --jobserver-fds=3D[[M,N]] +[ interface with GNU make jobserver; do not use manually] -k, --keywords=3DKEYWORDS [ select the tests matching all the comma-separated KEYWOR= DS] [ multiple \`-k' accumulate; prefixed \`!' negates a KEYWO= RD] @@ -974,6 +1010,11 @@ BEGIN { FS=3D"=07" } AS_ERROR([cannot create test line number cache]) rm -f "$at_suite_dir/at-source-lines" =20 +# Ignore --jobs with --jobserver-fds. +if test -n "$at_fd_in" && test -n "$at_fd_out"; then + at_jobs=3D0 +fi + # If parallel mode, don't output banners, don't split summary lines. if test $at_jobs -ne 1; then at_print_banners=3Dfalse @@ -1146,7 +1187,35 @@ trap 'exit_status=3D$? exit $exit_status' 1 2 13 15 at_first=3D: =20 -if test $at_jobs -ne 1 && +if test -n "$at_fd_in" && test -n "$at_fd_out"; then + # GNU make jobserver with bash or ksh. + echo + set x # We have one token initially. + for at_group in $at_groups; do + # Start one test group. + eval '( + at_func_group_prepare + if cd "$at_group_dir" && + at_func_test $at_group && + . "$at_test_source" # $at_fd_out>&- + then :; else + AS_WARN([unable to parse test group: $at_group]) + at_failed=3D: + fi + at_func_group_postprocess + AS_ECHO_N([+]) >&$at_fd_out + )'" $at_fd_in<&- &" + # Consume a token unless we have infinite jobs. + shift + if test $[@%:@] -gt 0; then :; else + read -n1 at_token <&$at_fd_in || break + set x $[*] + fi + test -f "$at_stop_file" && break + at_first=3Dfalse + done + wait +elif test $at_jobs -ne 1 && rm -f "$at_job_fifo" && ( mkfifo "$at_job_fifo" ) 2>/dev/null && exec AT_JOB_FIFO_FD<> "$at_job_fifo" diff --git a/tests/Makefile.am b/tests/Makefile.am index 01acf3a..4f00aba 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -139,12 +139,31 @@ clean-local: rm -f *.tmp rm -f -r autom4te.cache =20 +parse_makeflags =3D \ + dry_run=3D; pflags=3D; first=3D:; \ + for flag in $(MAKEFLAGS) --; do \ + case $$flag in \ + --jobserver-fds=3D*) pflags=3D"$$pflags $$flag";; \ + -- | *=3D*) break;; \ + --*) ;; \ + *) \ + if $$first; then \ + case $$flag in *n*) dry_run=3D:;; esac; \ + fi;; \ + esac; \ + first=3Dfalse; \ + done + check-local: atconfig atlocal $(TESTSUITE) - $(SHELL) $(TESTSUITE) $(TESTSUITEFLAGS) + @$(parse_makeflags); \ + echo $(SHELL) $(TESTSUITE) $$pflags $(TESTSUITEFLAGS); \ + $$dry_run $(SHELL) $(TESTSUITE) $$pflags $(TESTSUITEFLAGS) # $(MAKE) =20 # Run the test suite on the *installed* tree. installcheck-local: atconfig atlocal $(TESTSUITE) - $(SHELL) $(TESTSUITE) AUTOTEST_PATH=3D"$(bindir)" $(TESTSUITEFLAGS) + @$(parse_makeflags); \ + echo $(SHELL) $(TESTSUITE) $$pflags AUTOTEST_PATH=3D"$(bindir)" $(TESTSUI= TEFLAGS); \ + $$dry_run $(SHELL) $(TESTSUITE) $$pflags AUTOTEST_PATH=3D"$(bindir)" $(TE= STSUITEFLAGS) # $(MAKE) =20 =20 =20 diff --git a/tests/autotest.at b/tests/autotest.at index 54399f3..07172aa 100644 --- a/tests/autotest.at +++ b/tests/autotest.at @@ -834,6 +834,76 @@ AT_CHECK_AT_TEST([parallel errexit], [-j2 --errexit]) =20 =20 +AT_SETUP([parallel jobserver support]) + +AT_CHECK_AT_PREP([micro-suite], +[[AT_INIT([suite to test parallel execution]) +m4_for([count], [1], ]]AT_PARALLEL_TESTS_TOTAL[[, [], + [AT_SETUP([test number count]) + AT_CHECK([sleep 1]) + AT_CLEANUP +]) +]]) + +AT_DATA([Makefile], +[[TESTSUITE =3D ./micro-suite +parse_makeflags =3D \ + dry_run=3D; pflags=3D; first=3D:; \ + for flag in $(MAKEFLAGS) --; do \ + case $$flag in \ + --jobserver-fds=3D*) pflags=3D"$$pflags $$flag";; \ + -- | *=3D*) break;; \ + --*) ;; \ + *) \ + if $$first; then \ + case $$flag in *n*) dry_run=3D:;; esac; \ + fi;; \ + esac; \ + first=3Dfalse; \ + done + +# Note: we omit $(TESTSUITEFLAGS) here on purpose, +# in order not to be influenced by the outer testsuite. +check: + @$(parse_makeflags); \ + echo $(SHELL) $(TESTSUITE) $$pflags; \ + $$dry_run $(SHELL) $(TESTSUITE) $$pflags # $(MAKE) +]]) + +# Take care not to let the jobserver inherited by the toplevel +# testsuite invocation interfere. +MAKEFLAGS=3D +export MAKEFLAGS +: ${MAKE=3Dmake} + +# If make does not grok -jN, skip this test. +AT_CHECK([$MAKE -j[]AT_PARALLEL_TESTS_RUN check || exit 77], + [], [stdout], [ignore]) +# Ensure that all tests run, and lines are not split. +AT_CHECK([grep -c '^.\{53\}ok' stdout], [], [AT_PARALLEL_TESTS_TOTAL +]) + +# Ensure we really are faster than sequential execution; +# see the `parallel test execution' test for more details. +AT_CHECK([case `$MAKE --version 2>&1` in ]dnl + [ *GNU\ Make*) ]dnl + [ eval '( AS_ECHO_N([++]) | read -n1 at_token ) 2>/dev/null' ]dnl + [ || exit 77 ;; ]dnl + [ *) exit 77 ;; ]dnl + [esac]) +mkdir serial +sed 's|\./micro-suite|../micro-suite|g' < Makefile > serial/Makefile +AT_CHECK([$MAKE -j[]AT_PARALLEL_TESTS_RUN check & ]dnl + [sleep 3 && ]dnl + [cd serial && $MAKE check TESTSUITEFLAGS=3D-3 >/dev/null && ]dnl + [{ kill $! && exit 1; :; }], [], [stdout], [ignore]) +AT_CHECK([grep -c '^.\{53\}ok' stdout], [], [AT_PARALLEL_TESTS_TOTAL +]) +AT_CHECK([grep 'AT_PARALLEL_TESTS_TOTAL tests' stdout], [], [ignore]) + +AT_CLEANUP + + ## ------------------- ## ## srcdir propagation. ## ## ------------------- ## From MAILER-DAEMON Tue May 27 13:12:13 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K12ib-0008VT-J0 for mharc-autoconf-patches@gnu.org; Tue, 27 May 2008 13:12:13 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K12iZ-0008Uc-KH for autoconf-patches@gnu.org; Tue, 27 May 2008 13:12:11 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K12iX-0008TO-Gh for autoconf-patches@gnu.org; Tue, 27 May 2008 13:12:10 -0400 Received: from [199.232.76.173] (port=57861 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K12iX-0008TK-E3 for autoconf-patches@gnu.org; Tue, 27 May 2008 13:12:09 -0400 Received: from main.gmane.org ([80.91.229.2]:59496 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1K12iW-0002Ek-Ts for autoconf-patches@gnu.org; Tue, 27 May 2008 13:12:09 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1K12iQ-00079S-JQ for autoconf-patches@gnu.org; Tue, 27 May 2008 17:12:02 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 27 May 2008 17:12:02 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 27 May 2008 17:12:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Tue, 27 May 2008 17:11:50 +0000 (UTC) Lines: 59 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: shell redirection documentation X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2008 17:12:12 -0000 This bit me in the M4 testsuite (tests of behavior on closed fd's failed on Solaris 8 since the output leaked to the wrong fd, rather than generating the expected invalid redirection error), so I'm documenting it. >From f15bc5b04c7acc65f321ed3875e741000d0ce409 Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Tue, 27 May 2008 11:07:01 -0600 Subject: [PATCH] Document Solaris /bin/sh redirection pitfall. * doc/autoconf.texi (File Descriptors): Mention redirection bug. Signed-off-by: Eric Blake --- ChangeLog | 5 +++++ doc/autoconf.texi | 13 +++++++++++++ 2 files changed, 18 insertions(+), 0 deletions(-) diff --git a/ChangeLog b/ChangeLog index 15adde0..5d2c610 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,8 @@ +2008-05-27 Eric Blake + + Document Solaris /bin/sh redirection pitfall. + * doc/autoconf.texi (File Descriptors): Mention redirection bug. + 2008-05-14 Eric Blake Improve documentation of ! issues. diff --git a/doc/autoconf.texi b/doc/autoconf.texi index ee1ede5..cef2d3b 100644 --- a/doc/autoconf.texi +++ b/doc/autoconf.texi @@ -12698,6 +12698,19 @@ $ Within the process which runs the @samp{descrips} script, file descriptor 5 is closed. +Don't rely on redirection to a closed file descriptor to cause an +error. With Solaris @command{/bin/sh}, when the redirection fails, the +output goes to the original file descriptor. + +@example +$ @kbd{bash -c 'echo hi >&3' 3>&-; echo $?} +bash: 3: Bad file descriptor +1 +$ @kbd{/bin/sh -c 'echo hi >&3' 3>&-; echo $?} +hi +0 +@end example + @acronym{DOS} variants cannot rename or remove open files, such as in @samp{mv foo bar >foo} or @samp{rm foo >foo}, even though this is perfectly portable among Posix hosts. -- 1.5.5.1 From MAILER-DAEMON Thu May 29 09:18:38 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K1i1e-0008RA-J1 for mharc-autoconf-patches@gnu.org; Thu, 29 May 2008 09:18:38 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K1i1c-0008QB-NZ for autoconf-patches@gnu.org; Thu, 29 May 2008 09:18:36 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K1i1b-0008PV-Vc for autoconf-patches@gnu.org; Thu, 29 May 2008 09:18:36 -0400 Received: from [199.232.76.173] (port=46714 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K1i1b-0008PJ-Ln for autoconf-patches@gnu.org; Thu, 29 May 2008 09:18:35 -0400 Received: from [76.96.30.48] (port=41162 helo=QMTA05.emeryville.ca.mail.comcast.net) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K1i1a-0007qA-So for autoconf-patches@gnu.org; Thu, 29 May 2008 09:18:35 -0400 Received: from OMTA10.emeryville.ca.mail.comcast.net ([76.96.30.28]) by QMTA05.emeryville.ca.mail.comcast.net with comcast id XQcY1Z0060cQ2SLA500u00; Thu, 29 May 2008 12:54:59 +0000 Received: from [192.168.0.103] ([67.166.125.73]) by OMTA10.emeryville.ca.mail.comcast.net with comcast id XQuy1Z00B1b8C2B8W00000; Thu, 29 May 2008 12:54:59 +0000 X-Authority-Analysis: v=1.0 c=1 a=xe8BsctaAAAA:8 a=3f3fGZ3Mwy2ycoZ5pRkA:9 a=LL0lJ4vtOCA3yON-WPkA:7 a=TrjM8ns4PYtuobd1OUAXFvQ01iIA:4 a=eDFNAWYWrCwA:10 a=rPt6xJ-oxjAA:10 Message-ID: <483EA78D.5040207@byu.net> Date: Thu, 29 May 2008 06:54:37 -0600 From: Eric Blake User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080421 Thunderbird/2.0.0.14 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: autoconf-patches@gnu.org References: <20080526054742.GA11702@ins.uni-bonn.de> In-Reply-To: <20080526054742.GA11702@ins.uni-bonn.de> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized. X-Greylist: delayed 1413 seconds by postgrey-1.27 at monty-python; Thu, 29 May 2008 09:18:33 EDT Subject: Re: parallel autotest [0/3] X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 13:18:37 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 According to Ralf Wildenhues on 5/25/2008 11:47 PM: | There are many possible ways to parallelize Autotest testsuite | execution, for example: I'm still looking for time to review this series in depth, but appreciate the work you've put into it so far. | | a) have each test group be a 'make' target in a makefile, then use | parallel make. | b) leverage the job server from GNU make, | c) reimplement in shell a simplified job server a la parallel GNU make, | d) implement a "worker thread" parallelization in shell. | | Features and differences of these approaches: | | - (a) and (b) need GNU make for parallel execution, | - AFAICS (b) currently needs a shell that understands 'read -n1', | - (d) differs from (c) in that one subprocess executes more than one | test group, thus is potentially faster because it forks less, | - (b) has the nice feature that it allows to parallelize across multiple | test suites, and across testing and other, independent build activity. | That means, while (c) and (d) allow | make check TESTSUITEFLAGS='-j3' | to speed up things, (b) allows | gmake -j3 check | to profit. Interesting trades. I like that (c) and (d) can do parallel execution when the testsuite is run manually (without make); on the other hand, since make is usually the driver, I would tend to favor a solution along the lines of (b) that lets the testsuite work alongside other processes. | | I've experimented a bit with these approaches. I did not see an easy | way to get (a) to work under the restrictions that it may not start the | complete testsuite anew for each job: this has both very high overhead, | and/or it requires that user-provided startup bits like atlocal be | idempotent. I must confess that I didn't try very hard, though. The goal of still shipping a single 'testsuite' file that contains everything needed to create the multiple tests is nice. I agree that blindly calling 'testsuite n' for n parallel tests is too much overhead. On the other hand, it would sure be nice to call testsuite once with the user's TESTSUITEFLAGS to generate the subset of individual files to run, then turn around and run those individual files in parallel. Is there a way to write a Makefile include fragment which gets included if we detect GNU make, but is portably ignored for other makes, where we can then exploit gmake features to make parallel execution easier? | | This patch series consists of | 1) a preparatory patch refactoring the driver loop, This one is probably okay, even without the others. More comments to follow in that mail. | 2) an experimental patch implementing (c) if the system has mkfifo, | 3) an *experimental* patch supporting (b) if GNU make is used, and the | shell can 'read -n1'. I need more time to read through these. | | The patches (2) and (3) currently both have | - an unknown number of remaining race conditions, ;-) | - known file descriptor leaks to test grous (not sure whether to view | that as a problem or not), | - the bug^Wlimitation that, when a parallel run is interrupted, the | currently running test groups may still finish (see below also), Autotest behavior when using ^C is already fishy (I often find that it makes a test report as OK rather than failed, because it kills the test group before any failure file could be created). In other words, work to improve signal handling within autotest is useful independently of this patch series. | - tests that are probably still too strict. | | Right now, the only system where I had significant problems was Cygwin | with its seemingly limited named fifo emulation. I expect that, given | sufficient interest, somebody will fix this for me. ;-) Were you testing with cygwin 1.5.25 or the experimental cygwin 1.7.0? I agree that that the 1.5.x named fifos don't always work reliably. I'll certainly try to play more with this. - -- Don't work too hard, make some time for fun as well! Eric Blake ebb9@byu.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkg+p4wACgkQ84KuGfSFAYA3OgCfaFi1FoIrE6aeHmdOSIEC9H1I ViUAoIGgcDesTav5mO/USn65S+2/14J2 =bNWj -----END PGP SIGNATURE----- From MAILER-DAEMON Thu May 29 12:05:26 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K1kd4-0005Ed-Mo for mharc-autoconf-patches@gnu.org; Thu, 29 May 2008 12:05:26 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K1kd2-0005By-HD for autoconf-patches@gnu.org; Thu, 29 May 2008 12:05:24 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K1kd1-0005Af-B7 for autoconf-patches@gnu.org; Thu, 29 May 2008 12:05:23 -0400 Received: from [199.232.76.173] (port=54228 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K1kd1-0005AS-51 for autoconf-patches@gnu.org; Thu, 29 May 2008 12:05:23 -0400 Received: from main.gmane.org ([80.91.229.2]:32897 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1K1kd0-0000GB-GX for autoconf-patches@gnu.org; Thu, 29 May 2008 12:05:22 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1K1kcw-0003Qe-WD for autoconf-patches@gnu.org; Thu, 29 May 2008 16:05:19 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 May 2008 16:05:18 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 May 2008 16:05:18 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Thu, 29 May 2008 16:05:12 +0000 (UTC) Lines: 158 Message-ID: References: <20080526054742.GA11702@ins.uni-bonn.de> <20080526055004.GB11702@ins.uni-bonn.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Re: parallel autotest [1/3]: Refactor testsuite driver loop. X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 16:05:24 -0000 Ralf Wildenhues gmx.de> writes: > > 2008-04-13 Ralf Wildenhues gmx.de> > > * lib/autotest/general.m4 (AS_MESSAGE_LOG_FD): Move definition > earlier in the file. >... This patch is worth applying, once these nits are addressed... > > # The file containing the suite. > at_suite_log=$at_dir/$as_me.log > -# The file containing the location of the last AT_CHECK. > -at_check_line_file=$at_suite_dir/at-check-line > -# The file containing the exit status of the last command. > -at_status_file=$at_suite_dir/at-status > -# The files containing the output of the tested commands. > -at_stdout=$at_suite_dir/at-stdout > -at_stder1=$at_suite_dir/at-stder1 > -at_stderr=$at_suite_dir/at-stderr > -# The file containing the function to run a test group. > -at_test_source=$at_suite_dir/at-test-source > -# The file containing dates. > -at_times_file=$at_suite_dir/at-times > +# The directory containing helper files per test group. > +at_helper_dir=$at_suite_dir/at-groups You know, at this point, an ascii tree diagram might be a helpful comment (probably with # and not dnl, so that the comment is also present in the generated testsuite). Something like: testsuite - the testsuite testsuite.log - summarizes the complete testsuite run testsuite.dir/ - created during a run, remains after -d or failed test + at-groups/ - during a run: holds status of all groups in run | + nnn/ - during a run: holds meta-data about a group | | + at-check-line - during a group: holds location of current AT_CHECK | | + at-status - during a group: holds exit status of current AT_CHECK | | + at-stdout - during a group: captures stdout of current AT_CHECK | | + at-stder1 - during a group: captures stderr, including trace | | + at-stderr - during a group: captured stderr, with trace filtered out | | + at-test-source - during a group: portion of testsuite that defines group | | + at-times - during a group: holds timestamps for computing duration | | + pass - during a group: created if group passed | | + xpass - during a group: created if group xpassed | | + fail - during a group: created if group failed | | + xfail - during a group: created if group xfailed | | + skip - during a group: created if group skipped + at-stop - during a run: created to abort the run + at-source-lines - during a run: caches group line numbers + nnn/ - created for each group, remains after -d or failed test | + testsuite.log - summarizes the group results | + ... - files created during the group > +# Stop file: if it exists, do not start new jobs. > +at_stop_file=$at_suite_dir/at-stop Looks like a good idea. Please mention it in the ChangeLog entry. > @@ -959,19 +947,47 @@ BEGIN { FS="" } > test = substr ($ 0, 10) > print "at_sed" test "=\"1," start "d;" (NR-1) "q\"" > if (test == "'"$at_group"'") exit > -}' "$at_myself" > "$at_test_source" && > -. "$at_test_source" || > +}' "$at_myself" > "$at_suite_dir/at-source-lines" && > +. "$at_suite_dir/at-source-lines" || > AS_ERROR([cannot create test line number cache]) > +rm -f "$at_suite_dir/at-source-lines" I like the rename from $at_test_source to the literal at-source-lines, especially since the comment on the former no longer matched reality after we retooled things to use it as the line-number cache. > + > +# at_func_group_prepare > +# --------------------- > +# Prepare running a test group > +at_func_group_prepare () > +{ > + # The directory for additional per-group helper files. > + at_job_dir=$at_helper_dir/$at_group > + # The file containing the location of the last AT_CHECK. > + at_check_line_file=$at_job_dir/at-check-line Do we still need the at- prefix on files that live under $at_helper_dir? It seems a bit inconsistent to have $at_job_dir/at-stdout vs. $at_job_dir/pass, but I'm not sure whether the latter should be at-pass or whether we can simplify the former to stdout. > if test ! -f "$at_check_line_file"; then > - sed "s/^ */$as_me: warning: /" <<_ATEOF > - A failure happened in a test group before any test could be > - run. This means that test suite is improperly designed. Please > - report this failure to . > + sed "s/^ */$as_me: WARNING: /" <<_ATEOF Are we consistent on capitalization of warning: vs. WARNING:? > + echo "$at_res" > "$at_job_dir/$at_res" So both the name of the file and the contents of the file describe the result of the test. > > +# Wrap up the test suite with summary statistics. > +cd "$at_helper_dir" > + > +at_pass_list=`echo */pass | sed 's,/pass,,g; s,\*,,'` But here, you are only using the file name, and with several forks to compute the summary. What about these alternatives? 1. Keep the file name as the result, but the contents become the group id. This avoids a pipe, but still forks: echo "$at_group" > "$at_job_dir/$at_res" ... at_pass_list=`cat /dev/null */pass 2>/dev/null` 2. Keep the file name as the result but with an exploitable prefix, contents are irrelevant. Use a shell loop to compute the lists. Collecting the summary avoids forks altogether if we can then use an XSI construct, but is more expensive on deficient shells (implementation of the two versions of at_func_dirname left to the reader, but see libtool for hints): touch "$at_job_dir/result-$at_res" ... at_pass_list= ... for at_result in */result-*; do at_func_dirname $at_result at_group=$at_func_dirname_result case $at_result in *-pass) at_pass_list="$at_pass_list $at_group" ;; ... esac done 3. Like 2, but rework the loop variable to avoid needing at_func_dirname: touch "$at_job_dir/result-$at_res" ... at_pass_list= ... for at_group in $at_groups; do case $at_group/result-* in *-pass) at_pass_list="$at_pass_list $at_group" ;; ... esac done -- Eric Blake From MAILER-DAEMON Thu May 29 17:42:59 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K1ptj-0006cr-Ec for mharc-autoconf-patches@gnu.org; Thu, 29 May 2008 17:42:59 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K1pth-0006Zo-1z for autoconf-patches@gnu.org; Thu, 29 May 2008 17:42:57 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K1ptg-0006Yy-7H for autoconf-patches@gnu.org; Thu, 29 May 2008 17:42:56 -0400 Received: from [199.232.76.173] (port=54650 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K1ptf-0006Yn-Tn for autoconf-patches@gnu.org; Thu, 29 May 2008 17:42:55 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:60194) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K1ptf-0001Ch-9L for autoconf-patches@gnu.org; Thu, 29 May 2008 17:42:55 -0400 Received: from localhost.localdomain (xdsl-87-78-91-250.netcologne.de [87.78.91.250]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id 36E4940001D20; Thu, 29 May 2008 23:42:53 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1K1ptP-0008I5-KL; Thu, 29 May 2008 23:42:39 +0200 Date: Thu, 29 May 2008 23:42:39 +0200 From: Ralf Wildenhues To: Eric Blake Message-ID: <20080529214238.GA31816@ins.uni-bonn.de> Mail-Followup-To: Eric Blake , autoconf-patches@gnu.org References: <20080526054742.GA11702@ins.uni-bonn.de> <483EA78D.5040207@byu.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <483EA78D.5040207@byu.net> Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Cc: autoconf-patches@gnu.org Subject: Re: parallel autotest [0/3] X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 21:42:57 -0000 Hi Eric, * Eric Blake wrote on Thu, May 29, 2008 at 02:54:37PM CEST: > According to Ralf Wildenhues on 5/25/2008 11:47 PM: > | There are many possible ways to parallelize Autotest testsuite > | execution, for example: > > I'm still looking for time to review this series in depth, but appreciate > the work you've put into it so far. Take your time. This has been simmering here for several months, there's no reason to rush it in. > | a) have each test group be a 'make' target in a makefile, then use > | parallel make. > | b) leverage the job server from GNU make, > | c) reimplement in shell a simplified job server a la parallel GNU make, > | d) implement a "worker thread" parallelization in shell. > | > | Features and differences of these approaches: > | > | - (a) and (b) need GNU make for parallel execution, I should clarify: (a) is likely to cope with parallel non-GNU make, while (b) will ignore non-GNU make parallelism. For example, BSD makes typically allow parallel execution as well, but do not implement a job server. > | - AFAICS (b) currently needs a shell that understands 'read -n1', > | - (d) differs from (c) in that one subprocess executes more than one > | test group, thus is potentially faster because it forks less, > | - (b) has the nice feature that it allows to parallelize across multiple > | test suites, and across testing and other, independent build activity. > | That means, while (c) and (d) allow > | make check TESTSUITEFLAGS='-j3' > | to speed up things, (b) allows > | gmake -j3 check > | to profit. > > Interesting trades. I like that (c) and (d) can do parallel execution > when the testsuite is run manually (without make); Good point. I would not want to have (b) without any of (c), (d); rather, I thought of adding (c) and maybe also (b). > on the other hand, > since make is usually the driver, I would tend to favor a solution along > the lines of (b) that lets the testsuite work alongside other processes. Well, (b) kind of puts './testsuite' on par with 'gmake', as it then acts as distributor of work just like parallel (GNU) make does. > | I've experimented a bit with these approaches. I did not see an easy > | way to get (a) to work under the restrictions that it may not start the > | complete testsuite anew for each job: this has both very high overhead, > | and/or it requires that user-provided startup bits like atlocal be > | idempotent. I must confess that I didn't try very hard, though. > > The goal of still shipping a single 'testsuite' file that contains > everything needed to create the multiple tests is nice. I agree that > blindly calling 'testsuite n' for n parallel tests is too much overhead. > On the other hand, it would sure be nice to call testsuite once with the > user's TESTSUITEFLAGS to generate the subset of individual files to run, > then turn around and run those individual files in parallel. I don't quite understand what the last sentence in this paragraph is supposed to mean. In each of (b), (c), you can take any TESTSUITEFLAGS you would currently use; with (c), add '-j3' to TESTSUITEFLAGS, with (b), add '-j3' to the gmake command line directly. > Is there a way to write a Makefile include fragment which gets included if > we detect GNU make, but is portably ignored for other makes, where we can > then exploit gmake features to make parallel execution easier? As I understand this question, the at_parse_makeflags snippet shown in patch 3/3 does exactly that: it should be a no-op for non-GNU parallel make, as none of them use '--jobserver-fds='. If your question is about a general way to include something for GNU make only, one possibility is to have a GNUmakefile which includes Makefile plus extra, GNU make-specific code. > | The patches (2) and (3) currently both have > | - an unknown number of remaining race conditions, ;-) > | - known file descriptor leaks to test grous (not sure whether to view > | that as a problem or not), > | - the bug^Wlimitation that, when a parallel run is interrupted, the > | currently running test groups may still finish (see below also), > > Autotest behavior when using ^C is already fishy (I often find that it > makes a test report as OK rather than failed, because it kills the test > group before any failure file could be created). In other words, work to > improve signal handling within autotest is useful independently of this > patch series. Fully agreed; I've seen such fishy behavior, too, but mostly ignored it up to now. I fear that the behavior depends quite a bit on the sh and make implementations involved. > | Right now, the only system where I had significant problems was Cygwin > | with its seemingly limited named fifo emulation. I expect that, given > | sufficient interest, somebody will fix this for me. ;-) > > Were you testing with cygwin 1.5.25 or the experimental cygwin 1.7.0? I > agree that that the 1.5.x named fifos don't always work reliably. I'll > certainly try to play more with this. I think 1.5.25. My usual procedure is to boot w32, fire up cygwin setup.exe, run its update process without changing any settings manually, then do testing. I assume that, for experimental 1.7.0 I'd have to change some setting, no? FYI, here's some quick timings on an 8-way GNU/Linux system for -jP, Autoconf's testsuite, build tree kept on tmpfs, timings in seconds, efficiency in percent of the sequential case: P (b) eff (c) eff -------------------------------------- 1 435.5 100 2 224.2 97 224.5 97 4 118.5 92 118.6 92 7 78.8 79 78.9 79 8 73.7 74 73.7 74 Not very exciting. I think the drop-off in efficiency is mostly due to the inefficiency in Autotest: because it's a shell script that forks a lot, including many command pipes, and does a lot of I/O, many duties that benefit from extra processors that can do them in parallel. Other factors are of course inherently serial parts (Amdahl, the scheduling of the job server process), and, to a smaller degree, the load imbalance of the different tests. Running with a build tree on disk or even NFS has a large impact on overhead and parallel efficiency (roughly 35% over tmpfs in the sequential case, and it gets a lot worse with more processes). Cheers, Ralf From MAILER-DAEMON Thu May 29 17:45:24 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K1pw3-000724-Uq for mharc-autoconf-patches@gnu.org; Thu, 29 May 2008 17:45:24 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K1pw2-00071z-Gn for autoconf-patches@gnu.org; Thu, 29 May 2008 17:45:22 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K1pw0-00071n-Qb for autoconf-patches@gnu.org; Thu, 29 May 2008 17:45:21 -0400 Received: from [199.232.76.173] (port=54686 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K1pw0-00071k-JL for autoconf-patches@gnu.org; Thu, 29 May 2008 17:45:20 -0400 Received: from merkur.ins.uni-bonn.de ([131.220.223.13]:60396) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K1pvz-0001Wf-Al for autoconf-patches@gnu.org; Thu, 29 May 2008 17:45:20 -0400 Received: from localhost.localdomain (xdsl-87-78-91-250.netcologne.de [87.78.91.250]) by merkur.ins.uni-bonn.de (Postfix) with ESMTP id 5DB18400014EA; Thu, 29 May 2008 23:45:18 +0200 (CEST) Received: from ralf by localhost.localdomain with local (Exim 4.63) (envelope-from ) id 1K1pvl-0008IF-3S; Thu, 29 May 2008 23:45:05 +0200 Date: Thu, 29 May 2008 23:45:05 +0200 From: Ralf Wildenhues To: Eric Blake Message-ID: <20080529214504.GB31816@ins.uni-bonn.de> Mail-Followup-To: Eric Blake , autoconf-patches@gnu.org References: <20080526054742.GA11702@ins.uni-bonn.de> <20080526055004.GB11702@ins.uni-bonn.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Organization: Department of Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 3) Cc: autoconf-patches@gnu.org Subject: Re: parallel autotest [1/3]: Refactor testsuite driver loop. X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 21:45:22 -0000 Hi Eric, * Eric Blake wrote on Thu, May 29, 2008 at 06:05:12PM CEST: > Ralf Wildenhues gmx.de> writes: > > > > * lib/autotest/general.m4 (AS_MESSAGE_LOG_FD): Move definition > > earlier in the file. > This patch is worth applying, once these nits are addressed... > > +# The directory containing helper files per test group. > > +at_helper_dir=$at_suite_dir/at-groups > > You know, at this point, an ascii tree diagram might be a helpful comment > (probably with # and not dnl, so that the comment is also present in the > generated testsuite). Something like: *snip* Good point, I'll add something like that. > > +# Stop file: if it exists, do not start new jobs. > > +at_stop_file=$at_suite_dir/at-stop > > Looks like a good idea. Please mention it in the ChangeLog entry. No problem. I didn't mean to add it as public interface though, not sure if that's what you intend it to be. > > -}' "$at_myself" > "$at_test_source" && > > -. "$at_test_source" || > > +}' "$at_myself" > "$at_suite_dir/at-source-lines" && > > +. "$at_suite_dir/at-source-lines" || > I like the rename from $at_test_source to the literal at-source-lines, > especially since the comment on the former no longer matched reality after we > retooled things to use it as the line-number cache. Yep. > > +# at_func_group_prepare > > +# --------------------- > > +# Prepare running a test group > > +at_func_group_prepare () > > +{ > > + # The directory for additional per-group helper files. > > + at_job_dir=$at_helper_dir/$at_group > > + # The file containing the location of the last AT_CHECK. > > + at_check_line_file=$at_job_dir/at-check-line > > Do we still need the at- prefix on files that live under $at_helper_dir? No, I think we don't. > It > seems a bit inconsistent to have $at_job_dir/at-stdout vs. $at_job_dir/pass, > but I'm not sure whether the latter should be at-pass or whether we can > simplify the former to stdout. I think we should be able to simplify. However, we can do that in a separate patch, I think. Or right away, I don't mind. > > if test ! -f "$at_check_line_file"; then > > - sed "s/^ */$as_me: warning: /" <<_ATEOF > > - A failure happened in a test group before any test could be > > - run. This means that test suite is improperly designed. Please > > - report this failure to . > > + sed "s/^ */$as_me: WARNING: /" <<_ATEOF > > Are we consistent on capitalization of warning: vs. WARNING:? I think this change makes it consistent with the rest of Autoconf, yes. > > + echo "$at_res" > "$at_job_dir/$at_res" > > So both the name of the file and the contents of the file describe the result > of the test. Yep. I thought of echo dummy > "$at_job_dir/$at_res" bug figured that would look, erm, dumby. Apologies for the pun. ;-) > > +# Wrap up the test suite with summary statistics. > > +cd "$at_helper_dir" > > + > > +at_pass_list=`echo */pass | sed 's,/pass,,g; s,\*,,'` > > But here, you are only using the file name, and with several forks to compute > the summary. What about these alternatives? > > 1. Keep the file name as the result, but the contents become the group id. > This avoids a pipe, but still forks: > > echo "$at_group" > "$at_job_dir/$at_res" > ... > at_pass_list=`cat /dev/null */pass 2>/dev/null` This needs to read hundreds of files; which can easily be slower than a fork. For example, with warm caches, on NFS your command takes 100ms, while mine takes 8ms. The cold cache cases are a lot worse, of course. I'd be interested to hear whether things are different on w32. More generally, counting forks is not the only thing for efficiency; I/O is expensive, too. A minor point is that the above introduces newlines into the variables (no problem for pass but for xpass and the others). Of course, they can easily be killed on the go with set X $at_xpass_list; shift; at_xpass_count=$#; at_xpass_list=$* > 2. Keep the file name as the result but with an exploitable prefix, contents > are irrelevant. Use a shell loop to compute the lists. Collecting the summary > avoids forks altogether if we can then use an XSI construct, but is more > expensive on deficient shells (implementation of the two versions of > at_func_dirname left to the reader, but see libtool for hints): > > touch "$at_job_dir/result-$at_res" > ... > at_pass_list= > ... > for at_result in */result-*; do > at_func_dirname $at_result > at_group=$at_func_dirname_result > case $at_result in > *-pass) at_pass_list="$at_pass_list $at_group" ;; This scales quadratically in the number of passed tests. You can use func_append from Libtool, but only bash has +=. The fact that this forks once per test for slow shells makes this really unattractive. > ... > esac > done > 3. Like 2, but rework the loop variable to avoid needing at_func_dirname: > > touch "$at_job_dir/result-$at_res" > ... > at_pass_list= > ... > for at_group in $at_groups; do > case $at_group/result-* in The word after 'case' does not undergo filename expansion. That can be fixed, but still it would result in one filename expansion step be done for each test. I think that is slightly slower than doing just one filename expansion over all files. > *-pass) at_pass_list="$at_pass_list $at_group" ;; Still quadratic in number of passed tests. > ... > esac > done Cheers, Ralf From MAILER-DAEMON Thu May 29 18:28:09 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K1qbR-0007od-2X for mharc-autoconf-patches@gnu.org; Thu, 29 May 2008 18:28:09 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K1qbO-0007nE-O4 for autoconf-patches@gnu.org; Thu, 29 May 2008 18:28:06 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K1qbO-0007mt-C2 for autoconf-patches@gnu.org; Thu, 29 May 2008 18:28:06 -0400 Received: from [199.232.76.173] (port=60719 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K1qbN-0007mh-W6 for autoconf-patches@gnu.org; Thu, 29 May 2008 18:28:06 -0400 Received: from main.gmane.org ([80.91.229.2]:50501 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1K1qbN-0000B8-Kf for autoconf-patches@gnu.org; Thu, 29 May 2008 18:28:05 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1K1qbJ-0007j0-Dr for autoconf-patches@gnu.org; Thu, 29 May 2008 22:28:01 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 May 2008 22:28:01 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 May 2008 22:28:01 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Thu, 29 May 2008 22:27:54 +0000 (UTC) Lines: 110 Message-ID: References: <20080526054742.GA11702@ins.uni-bonn.de> <20080526055004.GB11702@ins.uni-bonn.de> <20080529214504.GB31816@ins.uni-bonn.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Re: parallel autotest [1/3]: Refactor testsuite driver loop. X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 22:28:07 -0000 Ralf Wildenhues gmx.de> writes: > > > +at_stop_file=$at_suite_dir/at-stop > > > > Looks like a good idea. Please mention it in the ChangeLog entry. > > No problem. I didn't mean to add it as public interface though, > not sure if that's what you intend it to be. No - we already document that anything in the at- filesystem namespace is reserved for internal workings. I merely wanted the string 'at_stop_file' in the ChangeLog, to make grepping for its introduction easier. > > > > Do we still need the at- prefix on files that live under $at_helper_dir? > > No, I think we don't. Then I'm okay with you dropping the at- prefix within the job-dirs. Note that my earlier comment about the at- filesystem namespace still applies to this situation: since the job-dirs live inside $at_helper_dir by the name at-groups, the various helper files are still off-limits to test groups by virtue of their parent directory. > > > It > > seems a bit inconsistent to have $at_job_dir/at-stdout vs. $at_job_dir/pass, > > but I'm not sure whether the latter should be at-pass or whether we can > > simplify the former to stdout. > > I think we should be able to simplify. However, we can do that in a > separate patch, I think. Or right away, I don't mind. At this point, keeping the commits separate is acceptable (one patch to refactor, another to rename at- files), since they are logically distinct changes. But it is your call if you want to squash those into one rebased patch, since all access to those helper files goes through the shell variables that your patch already touched. > > > > Are we consistent on capitalization of warning: vs. WARNING:? > > I think this change makes it consistent with the rest of Autoconf, yes. Aha. AS_MESSAGE does indeed print in all caps. I wonder, though, if that is a violation of GNU Coding Standards. At any rate, cleaning it up would be a separate followon patch, and shouldn't hold up the current commit. > > 1. Keep the file name as the result, but the contents become the group id. > > This avoids a pipe, but still forks: > > > > echo "$at_group" > "$at_job_dir/$at_res" > > ... > > at_pass_list=`cat /dev/null */pass 2>/dev/null` > > This needs to read hundreds of files; which can easily be slower than a > fork. For example, with warm caches, on NFS your command takes 100ms, > while mine takes 8ms. The cold cache cases are a lot worse, of course. > I'd be interested to hear whether things are different on w32. > > More generally, counting forks is not the only thing for efficiency; > I/O is expensive, too. Good point. I didn't really test, so I don't have numbers to back this up; I was just typing as I thought about it, trying to see if it was possible to get a speedup. Therefore, don't take this particular criticism as a rejection of your approach, so much as a thought experiment to see if there are any possible improvements (and there might not be). At any rate, your approach is already an improvement for non-XSI shells, as there are fewer expr calls to work around the lack of $(()). > > The fact that this forks once per test for slow shells makes > this really unattractive. I agree - 2 is a non-starter. > > 3. Like 2, but rework the loop variable to avoid needing at_func_dirname: > > > > touch "$at_job_dir/result-$at_res" > > ... > > at_pass_list= > > ... > > for at_group in $at_groups; do > > case $at_group/result-* in > > The word after 'case' does not undergo filename expansion. > That can be fixed, but still it would result in one filename > expansion step be done for each test. I think that is slightly > slower than doing just one filename expansion over all files. > > > *-pass) at_pass_list="$at_pass_list $at_group" ;; > > Still quadratic in number of passed tests. Can anybody think of a non-forking option 4? Use five 'test -f' instead of a 'case'? Otherwise, I'm inclined to concede that your approach is probably the fastest across all shells (only five list varibles to populate, so O(1) pathname expansions sed invocations regardless of number of test groups; and list creation via a single pathname expansion instead of quadratic variable reassignment). You are right that anything that has to read the files (rather than just using readdir() or access() to test for their existence) will probably be slower. On the other hand, since each file lives in a different directory, you are still hitting a lot of inodes to to the pathname expansion. -- Eric Blake From MAILER-DAEMON Thu May 29 18:56:31 2008 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1K1r2t-0003I6-8K for mharc-autoconf-patches@gnu.org; Thu, 29 May 2008 18:56:31 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K1r2r-0003GB-63 for autoconf-patches@gnu.org; Thu, 29 May 2008 18:56:29 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K1r2q-0003Fl-KZ for autoconf-patches@gnu.org; Thu, 29 May 2008 18:56:28 -0400 Received: from [199.232.76.173] (port=50673 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K1r2q-0003FT-B8 for autoconf-patches@gnu.org; Thu, 29 May 2008 18:56:28 -0400 Received: from main.gmane.org ([80.91.229.2]:38645 helo=ciao.gmane.org) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1K1r2p-0004Ap-KF for autoconf-patches@gnu.org; Thu, 29 May 2008 18:56:28 -0400 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1K1r2j-0000bv-RW for autoconf-patches@gnu.org; Thu, 29 May 2008 22:56:21 +0000 Received: from eblake.csw.l-3com.com ([128.170.36.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 May 2008 22:56:21 +0000 Received: from ebb9 by eblake.csw.l-3com.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 May 2008 22:56:21 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: autoconf-patches@gnu.org From: Eric Blake Date: Thu, 29 May 2008 22:56:14 +0000 (UTC) Lines: 123 Message-ID: References: <20080526054742.GA11702@ins.uni-bonn.de> <483EA78D.5040207@byu.net> <20080529214238.GA31816@ins.uni-bonn.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: main.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 128.170.36.124 (Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)) Sender: news X-detected-kernel: by monty-python.gnu.org: Linux 2.6, seldom 2.4 (older, 4) Subject: Re: parallel autotest [0/3] X-BeenThere: autoconf-patches@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Patches for autoconf - the GNU build system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 22:56:29 -0000 Ralf Wildenhues gmx.de> writes: > > | > > | - (a) and (b) need GNU make for parallel execution, > > I should clarify: (a) is likely to cope with parallel non-GNU make, > while (b) will ignore non-GNU make parallelism. For example, BSD makes > typically allow parallel execution as well, but do not implement a job > server. Yes, that helps me understand your intent - exposing the parallel tasks to any make's dependency engine, vs. hooking directly into the guts of GNU make's job server. > > > | - AFAICS (b) currently needs a shell that understands 'read -n1', > > | - (d) differs from (c) in that one subprocess executes more than one > > | test group, thus is potentially faster because it forks less, > > | - (b) has the nice feature that it allows to parallelize across multiple > > | test suites, and across testing and other, independent build activity. > > | That means, while (c) and (d) allow > > | make check TESTSUITEFLAGS='-j3' > > | to speed up things, (b) allows > > | gmake -j3 check > > | to profit. > > > > Interesting trades. I like that (c) and (d) can do parallel execution > > when the testsuite is run manually (without make); > > Good point. I would not want to have (b) without any of (c), (d); > rather, I thought of adding (c) and maybe also (b). I was hoping that you could combine the good parts of (b) and (c). > > The goal of still shipping a single 'testsuite' file that contains > > everything needed to create the multiple tests is nice. I agree that > > blindly calling 'testsuite n' for n parallel tests is too much overhead. > > On the other hand, it would sure be nice to call testsuite once with the > > user's TESTSUITEFLAGS to generate the subset of individual files to run, > > then turn around and run those individual files in parallel. > > I don't quite understand what the last sentence in this paragraph is > supposed to mean. Our current recommended formula for including an autotest suite in Makefile is to call it with TESTSUITEFLAGS as the user's hook to select a subset of tests. So I was trying to convey my thoughts on a flow that could be something like this: At configure time, config.status generates testsuite and also a makefile fragment that contains dependencies for each test's n.log, which is then included in the user's Makefile (automake supports makefile fragment inclusion even for non-GNU makes, or the fragment could be a multi-line AC_SUBST rather than an external file). At make time, each n.log depends on n.test and a first stamp file. Running 'testsuite --parallel-start TESTSUITEFLAGS=-10' creates the stamp file, creates an independent shell script n.test for tests 1-10 (each with much less startup overhead than testsuite; for example, it can be hardcoded to run with the better shell detected by testsuite, rather than having to repeat the gyrations of detecting a better shell itself), and a no-op n.test for the remaining test groups (merely touching n.log). After the stamp file exists, make can then run the various n.test files in parallel to create n.log files. Finally, a second stamp file depends on all of the n.log files; running 'testsuite --parallel-collect' gathers the results and creates the second stamp. That way, all you have to list for the dependency of check-local is the second stamp file. But that still doesn't answer the question of whether the startup costs for n shell scripts is reasonable, in relation to forking off the main driver shell script that has already collected a bunch of startup state. > > In each of (b), (c), you can take any TESTSUITEFLAGS you would currently > use; with (c), add '-j3' to TESTSUITEFLAGS, with (b), add '-j3' to the > gmake command line directly. And I suppose with (b) and (c) together, you could use: gmake -j3 check TESTSUITEFLAGS=-j3 but then you have two job servers both competing for processor load without communicating with each other on how to balance the load. > > > Is there a way to write a Makefile include fragment which gets included if > > we detect GNU make, but is portably ignored for other makes, where we can > > then exploit gmake features to make parallel execution easier? > > As I understand this question, the at_parse_makeflags snippet shown in > patch 3/3 does exactly that: it should be a no-op for non-GNU parallel > make, as none of them use '--jobserver-fds='. > > If your question is about a general way to include something for GNU > make only, one possibility is to have a GNUmakefile which includes > Makefile plus extra, GNU make-specific code. Whatever we decide, we should probably make it easy for package writers to call a single macro in configure.ac to generate the AC_SUBST that then spits out all the Makefile magic, rather than the current approach with several lines of documentation on how to edit Makefile to add a testsuite. For that matter, it might be worth installing /bin/autotest as a thin shell wrapper around /bin/autom4te --language=autotest. > Fully agreed; I've seen such fishy behavior, too, but mostly ignored it > up to now. I fear that the behavior depends quite a bit on the sh and > make implementations involved. And I think some of it stems from the fact that we base success on the contents of $at_status_file, which can easily be out-of-date if a test group is killed and no trap exists to update the file accordingly. > I think 1.5.25. My usual procedure is to boot w32, fire up cygwin > setup.exe, run its update process without changing any settings > manually, then do testing. I assume that, for experimental 1.7.0 > I'd have to change some setting, no? Your guess is right - since you haven't done anything special, you're running cygwin 1.5.25. -- Eric Blake