[Qemu-devel] ANNOUNCE: libguestfs 1.36 released

[Richard W.M. Jones]

I'm please to announce libguestfs 1.36, a library and a set of tools
for accessing and modifying virtual machine disk images.

This release represents about 7 months of work by many contributors.
I'd like to call out in particular substantial code contributions
from: Pino Toscano, Tomáš Golembiovský, Cédric Bosdonnat, Roman Kagan,
Dawid Zamirski and Matteo Cafasso.  Also our testing team: Tingting
Zheng, Xiang Hua Chen, Ming Xie, Junquin Zhou and Xiaodai Wang.

Amongst the new features are substantial enhancements to virt-v2v
including support for Debian and Ubuntu guest conversions, and
conversions from AWS; a new "virt-tail" command for following log
files in guests; even more support for encrypted guests; robustness
when handling Windows guests with corrupted registries; ExFAT support;
GCC 7 support; and considerable internal code refactoring to make
things more maintainable.

Of course dozens of other features have been added, and many bugs
fixed.  See the release notes below for full details.

You can get libguestfs 1.36 from here:

   Main website: http://libguestfs.org/
         Source: http://libguestfs.org/download/1.36-stable/
         Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=8391
 Debian/experimental: https://packages.debian.org/libguestfs0

Note for distro packagers: libmagic (part of the 'file' command line
tool) is now a required build dependency.  It was previously optional.

Rich.

----------------------------------------------------------------------

Release notes for libguestfs 1.36

These are also available online at:
http://libguestfs.org/guestfs-release-notes.1.html

   New features
     New tools

       Virt-tail is a new tool for following (tailing) log files within a
       guest, similar to the regular "tail -f" command.

     New features in existing tools

       Virt-customize, virt-get-kernel, virt-sparsify and virt-sysprep can now
       handle encrypted guests (Pino Toscano).

       Virt-builder and virt-customize now support --append-line, which is
       useful for adding lines to the end of configuration files.

       Virt-resize can now shrink and expand swap partitions (Pino Toscano).

       Virt-resize can now output to non-local disks (Pino Toscano).

       Virt-sysprep has a new operation called "backup-files" for removing
       editor backups, and "passwd-backups" for removing /etc/passwd- and
       similar.

       Virt-dib can now create checksums using the new --checksum option (Pino
       Toscano).

       Virt-dib can now create tgz files.  In addition, extended attributes
       and SELinux labels are preserved in the output tar file (Pino Toscano).

       Virt-dib can now create squashfs files (Pino Toscano).

       Bash tab-completion is now available in every command line tool that is
       shipped.

     virt-v2v and virt-p2v

       Conversions of Debian (6+) and Ubuntu (10.04+) guests are now supported
       (Tomáš Golembiovský, Pino Toscano).

       Conversions from SUSE Xen hypervisor have now been tested (Cédric
       Bosdonnat).

       Windows conversions may now install a program "pnp_wait.exe" which
       prevents conflicts between the Windows Plug-and-Play Manager and our
       own scripts that install virtio device drivers.  Enabling this requires
       the "pnp_wait.exe" program to be built separately (Roman Kagan).

       OVA files exported from AWS can now be converted (Shahar Havivi).

       When converting OVA files, in some circumstances virt-v2v can now read
       disk images directly from the OVA input file instead of needing to
       unpack the OVA file to a temporary directory.  This can save large
       amounts of disk space, and is quicker (Tomáš Golembiovský).

       Virt-p2v can now be built on RHEL 5 (2007-era) Linux and RHEL 6, which
       means that 32 bit and hardware with ancient fakeraid disks can be
       virtualized.  Binaries of virt-p2v based on these old versions of RHEL
       can be found at http://oirase.annexia.org/virt-p2v/

       Virt-p2v can now use nbdkit (an NBD server) as an alternative to qemu-
       nbd.  In addition, virt-p2v can use "socket activation" which is a more
       robust method for opening the NBD listening socket.  Socket activation
       works with qemu-nbd or nbdkit, but requires the most up to date
       versions.

       To prevent timeouts during P2V conversions, virt-p2v inhibits power
       saving on the physical machine, and also sends ping packets over the
       ssh control and data connections.

       Virt-v2v no longer removes the "Processor" and "Intelppm" nodes from
       the Windows Registry.

       Xen and vCenter conversions can now be done using the libvirt backend,
       provided libvirt ≥ 2.1.0 is used.

       /dev/srX (SCSI CD-ROM) devices are ignored (Tomáš Golembiovský).

       When converting SUSE Linux guests, use /etc/modprobe.conf.local if it
       exists.

       Removing VMware tools from Linux guests should now work reliably (Pino
       Toscano).

       When converting OVA files, virt-v2v now checks the disk image hash
       contained in the manifest file, whereas previously it was ignored
       because of a mistake in the code.  In addition, SHA256 hashes are now
       supported (Tomáš Golembiovský).

       When converting OVA files that contain compressed disk images, we now
       trust the "ovf:compression" attribute from the metadata instead of
       using content sniffing on the disk image (Tomáš Golembiovský).

       A new flag --vdsm-compat has been added to -o vdsm mode, allowing more
       efficient qcow2 images to be generated.

       Since the RHEV product [commercially supported version of oVirt] has
       now been renamed to RHV, "RHEV" was changed to "RHV" throughout the
       tooling.  In particular, -o rhev becomes -o rhv, although the old name
       can be used for backwards compatibility.

       The "RHEV-APT" utility is only installed when the output hypervisor is
       oVirt or RHV.

       A problem with virt-v2v running out of memory when doing an SELinux
       relabel of the guest filesystem should now be resolved.

     Language bindings

       Setting "EXTRA_JAVAC_FLAGS" before ./configure allows you to add
       arbitrary flags to "javac" when compiling the Java bindings.

       Use of any libguestfs API which returned a single struct, from Perl or
       Java bindings, would have leaked memory.  This has now been fixed (Pino
       Toscano).

     Inspection

       Inspection of Windows guests with mildly corrupted Windows Registry
       hives should now work instead of failing with an error.  This feature
       requires hivex ≥ 1.3.14 (Dawid Zamirski).

       For Debian guests, we can now read the URL, source name, summary and
       full description of installed packages.  For RPM-based guests we can
       read the URL, summary and description (Pino Toscano).

       Inspection of guest operating systems that use a separate /usr
       partition should now work more reliably (Pino Toscano).

       When parsing guest /etc/fstab, paths are reduced to a canonical form
       (eg. "///usr//local//" → "/usr/local").

       Inspection of btrfs subvolumes now works where the /etc/fstab options
       field contains commas.

       /dev/cdN devices in guest /etc/fstab are ignored (Pino Toscano).

     Architectures and platforms

       Stable releases are now tested on aarch64, ppc64 and ppc64le
       architectures.

       The RISC-V architecture is now handled as a guest, and libguestfs will
       now at least compile on RISC-V (but probably not work as there is no
       usable qemu at the time of writing).

       Detection of S/390 and S/390x binaries and guests is now supported.

     Other

       ExFAT filesystems are now supported (Miles Wolbe).

   Security
       See also guestfs-security(1).

       There were no CVEs reported in this development cycle.  However some
       security-related hardening was carried out as described below.

       Temporary filenames are now always generated using randomness from
       /dev/urandom (previously the C function random(3) was used in one
       case).

       The $TERM environment variable is now validated before passing it
       through to the appliance kernel command line.

   API
     New APIs

       "guestfs_aug_transform"
           Exposes the Augeas "aug_transform" API (Pino Toscano).

       "guestfs_find_inode"
           Find files by inode number (Matteo Cafasso).

       "guestfs_inspect_get_windows_software_hive"
       "guestfs_inspect_get_windows_system_hive"
           Return the path to the Windows "HKLM\SYSTEM" and "HKLM\SOFTWARE"
           hives computed during inspection.

       "guestfs_mksquashfs"
           Create a squashfs filesystem from a path (Pino Toscano).

     Other API changes

       "guestfs_add_domain"
           This call now handles libvirt file-based volumes correctly, where
           previously these would have been ignored.  Also handled are disks
           which require libvirt authentication secrets to open (Pino
           Toscano).

       "guestfs_canonical_device_name"
           This call will no longer incorrectly modify Linux software RAID
           device names (like /dev/mdX).

       "guestfs_file_architecture"
           Previously the "file_architecture" API could return either of the
           strings "i386" or "i486" for 32 bit x86 binaries.  It now only
           returns "i386" (as documented).

           This API can now return the following new values: "riscv32",
           "riscv64", "riscv128", "s390", "s390x".

       "guestfs_hivex_open"
           This now has an optional "GUESTFS_HIVEX_OPEN_UNSAFE" flag which
           allows certain corrupted Windows Registry hives to be opened.  This
           feature requires hivex ≥ 1.3.14 (Dawid Zamirski).

       "guestfs_list_partitions"
           This call now returns Linux software RAID partitions.

       "guestfs_part_to_dev"
           This call now correctly handles partition names which include
           "p<N>" (Pino Toscano).

       "guestfs_set_label"
           This call can now change the labels of swap partitions (Pino
           Toscano).

   Build changes
       libmagic, the library part of the "file" command, is now required at
       build time (previously optional).

       GCC 7 is now supported.

       "Silent rules" are now used for OCaml programs, Java bindings.  To show
       the full command line executed, add "V=1" on the make command line
       (Pino Toscano).

       Slow testing ("make check-slow") now covers: firstboot scripts in Linux
       guests; v2v conversion of a selection of real Linux guests; the virt-
       customize --hostname and --timezone settings; the --root-password
       parameter; that the serial console works in virt-builder guests.

       Large generated C source files, eg. the list of commands found in
       fish/cmds.c (and many more), have been split into smaller files to
       speed parallel compilation.

       "make maintainer-check-extra-dist" now checks that all generated files
       are included in the tarball.

       The tests no longer assume that "." is in Perl's @INC, as it is going
       to be removed soon (Pino Toscano).

       Debian hosts using UsrMerge are now supported (Pino Toscano).

       Header files and C structs can now have internal documentation using
       the special "/** ... */" comments.

       "@VAR@" subtitutions in "./run" are now fully quoted.  This is
       necessary so that (eg) "./configure PYTHON=/some/path" works robustly
       if "/some/path" contains characters that need to be quoted (Hilko
       Bengen).

       gperf ≥ 3.1 is now supported.

       Kraxel's old edk2 builds can no longer be used for UEFI support.  UEFI
       code is now fully free software, so use the versions bundled with your
       Linux distro instead.

       Virt-p2v can now be compiled on RHEL 5 (2007-era) Linux with Gtk 2.10.

   Internals
       The generator and mllib "Common_utils" modules are now shared from the
       same source file.

       A considerable amount of common code has been moved into the common
       directory in the source and is now compiled only once.  The mini-
       libraries located under here are: common/edit, common/errnostring,
       common/miniexpect, common/options, common/parallel, common/progress,
       common/protocol, common/utils, common/visit, common/windows.

       The directory containing the main library code has moved from src →
       lib.

       All tests written in shell script now use a common file of utility
       functions (tests/test-functions.sh).  There are several new utility
       functions, mainly for skipping tests.  Also these test scripts can now
       use autoconf-like path variables like $abs_top_srcdir.

       UEFI paths are now stored in the generator (generator/uefi.ml).

       The way the generator handles actions and procedure numbers was changed
       quite substantially.  See generator/actions_*.ml and
       generator/proc_nr.ml.

       The gnulib "getprogname" module is now used everywhere when
       needing/printing the program name (Pino Toscano).

       perl/Guestfs.c is not translatable (Nikos Skalkotos).

       Virt-builder templates moved from builder/website to builder/templates
       and there is now a single unified program which can build any template.

       All Windows registry utilities used by virt-customize and virt-v2v have
       been moved to a common module called "Registry" under mllib.

       All POSIX bindings have been moved to a new module called "Unix_utils"
       under mllib.

       Inspection, virt-customize and virt-v2v no longer recompute the Windows
       "%systemroot%", "CurrentControlSet" or paths to the "HKLM\SYSTEM" and
       "HKLM\SOFTWARE" hives in multiple places.  Instead these are all
       computed once (during inspection) and passed to the other tools through
       various "guestfs_inspect_get_windows_*" APIs.

       "/dev/pts" is now available inside the appliance, so any tools we run
       which require a pty will now work (Pino Toscano).

       Most OCaml warnings have been fixed.

       There is now a single common function for creating temporary files
       ("guestfs_int_make_temp_path") (Matteo Cafasso).

       The $TERM environment variable is now validated before passing it
       through to the appliance kernel command line.

       Useless USB and memballoon devices are no longer created in the
       appliance (Laine Stump).

       On aarch64 we now use virtio-pci for the appliance.  This is somewhat
       faster than virtio-mmio.

       Use of srandom(3) and random(3) has been minimized.  In particular,
       temporary filenames are no longer created based on randomness returned
       by random(3), but /dev/urandom is used instead.

   Bugs fixed
       https://bugzilla.redhat.com/1425306
           typo error in virt-tail man page

       https://bugzilla.redhat.com/1418283
           virt-v2v: appliance runs out of memory running setfiles command

       https://bugzilla.redhat.com/1417549
           /usr/bin/x86_64-linux-gnu-ld.bfd.real:
           ../common/progress/.libs/libprogress.a(libprogress_la-progress.o):
           undefined reference to symbol 'UP@@NCURSES_TINFO_5.0.19991023'

       https://bugzilla.redhat.com/1417444
           *** No rule to make target '../perl/lib/Sys/Guestfs.c', needed by
           'libguestfs.pot'

       https://bugzilla.redhat.com/1416941
           compile of 1.34.3 fails with gperf 3.1

       https://bugzilla.redhat.com/1414682
           guestfs_canonical_device_name incorrectly returns /dev/sd0 for MD
           devices (/dev/md0)

       https://bugzilla.redhat.com/1414510
           guestfs_list_filesystems does not recognize ddf partitions

       https://bugzilla.redhat.com/1409023
           [Debian] ldmtool not installed in the appliance

       https://bugzilla.redhat.com/1404287
           qemu-kvm cannot boot RHEL 7 kernel with TCG, hangs at "Probing EDD
           (edd=off to disable)..."

       https://bugzilla.redhat.com/1404182
           RFE: virt-resize should support a URL as the outdisk

       https://bugzilla.redhat.com/1401474
           Importing VMs from VMware is failing with error "Inspection field
           'i_arch' was 'unknown'"

       https://bugzilla.redhat.com/1401320
           RFE: Increate virt-sysprep coverage a bit

       https://bugzilla.redhat.com/1400205
           Add --vdsm-compat=1.1 flag for VDSM

       https://bugzilla.redhat.com/1398070
           typo error in man page

       https://bugzilla.redhat.com/1392798
           secrets from libvirt domains are not read

       https://bugzilla.redhat.com/1390876
           "--machine-readable" info should be updated in virt-v2v manual page

       https://bugzilla.redhat.com/1379289
           RFE: virt-p2v should support mnemonic operations

       https://bugzilla.redhat.com/1378022
           There is virt-v2v warning about <listen type='none'> during
           converting a guest which has listen type='none' in XML

       https://bugzilla.redhat.com/1377081
           virt-p2v manual should update the new dialog information

       https://bugzilla.redhat.com/1375157
           virt-v2v: -i ova: Permission denied when using libvirt and running
           as root

       https://bugzilla.redhat.com/1374651
           Can't install qxl driver for display device in win7 guest after
           converting to glance by virt-v2v

       https://bugzilla.redhat.com/1374405
           There is HTTP 404 error info when convert guest to glance by
           virt-v2v

       https://bugzilla.redhat.com/1374232
           selinux relabel fails on RHEL 6.2 guests with "libguestfs error:
           selinux_relabel: : Success"

       https://bugzilla.redhat.com/1372668
           Process status is not normal in windows guest after converted from
           kvm to rhev by virt-v2v

       https://bugzilla.redhat.com/1372269
           Builder does not set hostname properly for Debian 8 (Jessie)

       https://bugzilla.redhat.com/1371843
           Improve OVA import compatibility

       https://bugzilla.redhat.com/1370424
           virt-manager coredump when vm with gluster image exists

       https://bugzilla.redhat.com/1367839
           Cannot import VMs from Xen and VMware when using RHEL7.3 host.

       https://bugzilla.redhat.com/1367738
           Missing bash completion scripts for: virt-diff guestunmount virt-
           copy-in virt-copy-out virt-customize virt-get-kernel
           virt-p2v-make-disk virt-p2v-make-kickstart virt-tar-in virt-tar-out
           virt-v2v-copy-to-local virt-win-reg

       https://bugzilla.redhat.com/1367615
           OVMF file which is built for rhel7.3  can't be used for virt-v2v
           uefi conversion

       https://bugzilla.redhat.com/1366456
           Converting rhel7 host installed on RAID:warning: fstrim: fstrim:
           /sysroot/: the discard operation is not supported

       https://bugzilla.redhat.com/1366049
           RFE: libvirt backend: support handling disks stored as volume name
           in a pool

       https://bugzilla.redhat.com/1365005
           Guest name is incorrect if convert guest from disk image by
           virt-v2v

       https://bugzilla.redhat.com/1362649
           RFE: virt-sysprep does not utilize libguestfs encryption support

       https://bugzilla.redhat.com/1354507
           virt-v2v conversions from vCenter do not consistently obey the
           proxy environment variables

       https://bugzilla.redhat.com/1168144
           warning: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation
           not supported (ignored) when convert win2003 guest from xen server

       https://bugzilla.redhat.com/1161019
           RFE: Only install RHEV-APT if virt-v2v -o rhev/-o vdsm option is
           used

       https://bugzilla.redhat.com/1152369
           virt-v2v failed to convert RHEL 6.7 UEFI guest: no
           grub1/grub-legacy or grub2 configuration file was found

       https://bugzilla.redhat.com/1141631
           [RFE] virt-v2v should support convert a guest to a dir-pool with
           using pool's uuid

       https://bugzilla.redhat.com/1134878
           libvirt reports json "backing file" is missing

       https://bugzilla.redhat.com/1019388
           firstboot scripts (virt-builder, virt-sysprep) don't work for
           Debian 6 & 7 guests

       https://bugzilla.redhat.com/737600
           virt-v2v windows xp - machine dies BSOD - processr,sys - workaround
           provided


-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW