Re: [Qemu-devel] [PATCH] scripts: use git archive in archive-source

[Daniel P . Berrangé]

On Wed, Jan 30, 2019 at 07:55:30AM +0100, Gerd Hoffmann wrote:
> Use git archive to create tarballs of qemu and submodules instead of
> cloning the repository and the submodules.  This is a order of magnitude
> faster because it doesn't fetch the submodules from the internet each
> time the script runs.

There's another patch addressing the same problem here, which takes
the approach of 'git clone' on the submodules to avoid pulling them
from the internet each time:

  https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06588.html

> Signed-off-by: Gerd Hoffmann <address@hidden>
> ---
>  scripts/archive-source.sh | 46 ++++++++++------------------------------------
>  1 file changed, 10 insertions(+), 36 deletions(-)
> 
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index 6eed2a29bd..beae968f80 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -19,8 +19,7 @@ if test $# -lt 1; then
>  fi
>  
>  tar_file=$(realpath "$1")
> -list_file="${tar_file}.list"
> -vroot_dir="${tar_file}.vroot"
> +sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar")
>  
>  # We want a predictable list of submodules for builds, that is
>  # independent of what the developer currently has initialized
> @@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot"
>  # different to the host OS.
>  submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 
> tests/fp/berkeley-testfloat-3"
>  
> -trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 
> 3 15
> +trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15
>  
>  if git diff-index --quiet HEAD -- &>/dev/null
>  then
> @@ -36,38 +35,13 @@ then
>  else
>      HEAD=$(git stash create)
>  fi
> -git clone --shared . "$vroot_dir"
> -test $? -ne 0 && error "failed to clone into '$vroot_dir'"
> -
> -cd "$vroot_dir"
> -test $? -ne 0 && error "failed to change into '$vroot_dir'"
> -
> -git checkout $HEAD
> -test $? -ne 0 && error "failed to checkout $HEAD revision"
> -
> +git archive --format tar $HEAD > "$tar_file"
> +test $? -ne 0 && error "failed to archive qemu"
>  for sm in $submodules; do
> -    git submodule update --init $sm
> -    test $? -ne 0 && error "failed to init submodule $sm"
> +     git submodule update --init "$sm"

The current archive-source.sh code is careful to not touch the
developer's current git checkout, because it runs  submodule
update in a clone'd tree. The earlier patch hunk gets rid of the
clone, so now submodule update is touching the developer's master
git checkout. IMHO this is not acceptable - running tests (which
use archive-source.sh) must never alter the developer's git checkout
status in any way.

The alternate patch mentioned above does that.

Even if we keep the git clone part, we could still usefully
switch to running git archive, instead of re-inventing it
via git ls-files + tar.

> +     test $? -ne 0 && error "failed to update submodule $sm"
> +     (cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"
> +     test $? -ne 0 && error "failed to archive submodule $sm"
> +     tar --concatenate --file "$tar_file" "$sub_file"
> +     test $? -ne 0 && error "failed append submodule $sm to $tar_file"
>  done
> -
> -if test -n "$submodules"; then
> -    {
> -        git ls-files || error "git ls-files failed"
> -        for sm in $submodules; do
> -            (cd $sm; git ls-files) | sed "s:^:$sm/:"
> -            if test "${PIPESTATUS[*]}" != "0 0"; then
> -                error "git ls-files in submodule $sm failed"
> -            fi
> -        done
> -    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > 
> "$list_file"
> -else
> -    git ls-files > "$list_file"
> -fi
> -
> -if test $? -ne 0; then
> -    error "failed to generate list file"
> -fi
> -
> -tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
> -
> -exit 0
> -- 
> 2.9.3
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|