Re: Build Iterates Indefinitely on "Generating include/autoconf.mk"

[Grant Erickson]

On 3/26/08 4:52 PM, Grant Erickson wrote:
> While I generally develop against local file systems only, today I did a build
> against an NFS mounted file system and encountered new behavior in the u-boot
> build that I had never before seen.
> 
> That is, an indefinite iteration on "Generating include/autoconf.mk":
> 
>     $ CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" gmake -C
> u-boot-1.3.2/ O=${PWD}/.build/ haleakala_config
>     gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
>     Configuring for haleakala board...
>     gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
>     $ CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" gmake -C
> u-boot-1.3.2/ O=${PWD}/.build/ all
>     gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
>     Generating include/autoconf.mk
>     gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
>     gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
>     Generating include/autoconf.mk
>     gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
>     ...
> 
> This continues for 5, 10, 15 minutes or longer if I let it. Has anyone else
> seen this?
> 
> Thinking that this was a classic build machine / NFS server time
> synchronization disconnect, I confirmed that both the local machine and NFS
> server were slaved to the same NTP server and ran the following quick check:
> 
>     $ rm -f test; echo > test; ls -l --full-time test; date +'%T.%N'
>     -rw-r--r--  1 gerickson gerickson 1 2008-03-26 16:26:03.635637000 -0700
>     test
>     16:26:03.648288000
> 
> The skew seemed reasonable given inherent latencies in just executing the
> commands, so I ruled out time skew as an issue.
> 
> By complete coincidence, I then attempted to retest things in /tmp (tmpfs)
> rather than the usual /build (ext3) and got the same results as with NFS:
> 
> [ ... output deleted ... ]
> 
> I then decided to see if it was a make-specific issue and tested with
> make-3.79 rather than make-3.81. Sure enough, it worked in all locations
> (tmpfs, ext3, nfs).
> 
> I then repeated with make-3.80 and came up with the following matrix:
> 
>             make-3.79   make-3.80   make-3.81
>     -----------------------------------------
>     ext3    SUCCESS     SUCCESS     SUCCESS
>     nfs     SUCCESS     FAILURE     FAILURE
>     tmpfs   SUCCESS     FAILURE     FAILURE
>     -----------------------------------------
> 
> If someone else has encountered this, are there any work-arounds in the u-boot
> build that you've experimented with?

For those interested, I've done some additional research on this issue and
have come up with the results shown below. All experiments were carried out
on a i686-pc-linux-gnu 2.6.9-34.ELsmp machine. The '/usr/local/bin' versions
of make are vendor-supplied.

Summary
-------

    ------------------------------------------------------------------------
    Shell   Make                                     NFS     tmpfs    ext3
    ========================================================================
    bash    /usr/bin/make (3.80)                   FAILURE  FAILURE  SUCCESS
    bash    /usr/local/bin/make-3.79 (3.79)        SUCCESS  SUCCESS  SUCCESS
    bash    /usr/local/bin/make-3.80 (3.80)        SUCCESS  SUCCESS  SUCCESS
    bash    /usr/local/bin/make-3.81 (3.81)        SUCCESS  SUCCESS  SUCCESS
    bash    /usr/local/bin/gmake (3.81)            FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.79)            FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.80 w/ librt)   FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.80 w/o librt)  FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.81 w/ librt)   FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.81 w/o librt)  FAILURE  FAILURE  SUCCESS
    ========================================================================

Analysis
--------

The failure of two seemingly identical versions of make
(/usr/local/bin/make-3.81 vs. /usr/local/bin/gmake) offered a chance to
examine what differences might exist between the two versions.

The most notable difference is the libraries against which each is linked:

    % ldd /usr/bin/make
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/local/bin/make-3.79
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/local/bin/make-3.80
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/software/bin/gmake
            librt.so.1 => /lib/tls/librt.so.1 (0x00561000)
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00958000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/software/rats/bin/make-3.81
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

So, at first glance, it would appear that make 3.81's use of clock_gettime()
and the subsequent link against librt and, in turn, libpthread is somehow
related.

However, it turns out to not be as simple as that. If we look at the two
local, private builds of make-3.81 that I generated we find that the one
that uses clock_gettime() and librt as well as the one that does not BOTH
FAIL:

    % ldd ~/tmp/bin/make-librt
            librt.so.1 => /lib/tls/librt.so.1 (0x00561000)
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00958000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd ~/tmp/bin/make-nolibrt
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

Set-up
------

1) The 'bash' shell builds were invoked by first logging into my default
   shell, 'tcsh' and then invoking 'bash'.

2) I installed the DENX ELDK 4.1 to my home directory at ${HOME}/eldk/4.1.

3) The builds were performed by doing the following:

   $ for makebin in <Make to Invoke>; do rm -rf .build/*;
CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" ${makebin} -C
u-boot-1.3.2/ O=${PWD}/.build haleakala_config; CROSS_COMPILE=ppc_4xx-
PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" ${makebin} -C u-boot-1.3.2/
O=${PWD}/.build all; done

5) The local, private version of make (3.81) was created by:

   % ~/src/gnu/
   % wget http://ftp.gnu.org/gnu/make/make-3.81.tar.gz
   % cd source/
   % tar zxf ../make-3.81.tar.gz
   % cd ../build/
   % mkdir make-3.81
   % cd make-3.81/
   % ../../source/make-3.81/configure --prefix=${HOME}/tmp
   % make
   % make install

  and was built against gcc-4.1.1. The non-librt version was created by
  removing '#define HAVE_CLOCK_GETTIME 1' from config.h and removing '-lrt'
  from LIBS in Makefile.

Any insights on what might be at play? I thought perhaps GCC, but
recompiling all of the private versions against gcc-3.4.6 yields the same
results.

Regards,

Grant Erickson