[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v7 05/12] tests/vm: Added configuration file support
From: |
Alex Bennée |
Subject: |
Re: [PATCH v7 05/12] tests/vm: Added configuration file support |
Date: |
Fri, 22 May 2020 15:26:08 +0100 |
User-agent: |
mu4e 1.4.6; emacs 28.0.50 |
Robert Foley <address@hidden> writes:
> Changes to tests/vm/basevm.py to allow accepting a configuration file
> as a parameter. Allows for specifying VM options such as
> cpu, machine, memory, and arbitrary qemu arguments for specifying options
> such as NUMA configuration.
> Also added an example conf_example_aarch64.yml and conf_example_x86.yml.
>
> Signed-off-by: Robert Foley <address@hidden>
> Reviewed-by: Peter Puhov <address@hidden>
Reviewed-by: Alex Bennée <address@hidden>
> ---
> configure | 9 ++++++
> tests/vm/Makefile.include | 6 ++++
> tests/vm/basevm.py | 40 +++++++++++++++++++++++-
> tests/vm/conf_example_aarch64.yml | 51 +++++++++++++++++++++++++++++++
> tests/vm/conf_example_x86.yml | 50 ++++++++++++++++++++++++++++++
> 5 files changed, 155 insertions(+), 1 deletion(-)
> create mode 100644 tests/vm/conf_example_aarch64.yml
> create mode 100644 tests/vm/conf_example_x86.yml
>
> diff --git a/configure b/configure
> index a574250524..89d11aa5d4 100755
> --- a/configure
> +++ b/configure
> @@ -950,6 +950,13 @@ do
> fi
> done
>
> +# Check for existence of python3 yaml, needed to
> +# import yaml config files into vm-build.
> +python_yaml="no"
> +if $(python3 -c "import yaml" 2> /dev/null); then
> + python_yaml="yes"
> +fi
> +
> : ${smbd=${SMBD-/usr/sbin/smbd}}
>
> # Default objcc to clang if available, otherwise use CC
> @@ -6581,6 +6588,7 @@ if test "$docs" != "no"; then
> echo "sphinx-build $sphinx_build"
> fi
> echo "genisoimage $genisoimage"
> +echo "python_yaml $python_yaml"
> echo "slirp support $slirp $(echo_version $slirp $slirp_version)"
> if test "$slirp" != "no" ; then
> echo "smbd $smbd"
> @@ -7642,6 +7650,7 @@ echo "PYTHON=$python" >> $config_host_mak
> echo "SPHINX_BUILD=$sphinx_build" >> $config_host_mak
> echo "SPHINX_WERROR=$sphinx_werror" >> $config_host_mak
> echo "GENISOIMAGE=$genisoimage" >> $config_host_mak
> +echo "PYTHON_YAML=$python_yaml" >> $config_host_mak
> echo "CC=$cc" >> $config_host_mak
> if $iasl -h > /dev/null 2>&1; then
> echo "IASL=$iasl" >> $config_host_mak
> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
> index 74ab522c55..d9b34eae63 100644
> --- a/tests/vm/Makefile.include
> +++ b/tests/vm/Makefile.include
> @@ -43,6 +43,12 @@ endif
> @echo " V=1 - Enable verbose ouput on host
> and guest commands"
> @echo " QEMU=/path/to/qemu - Change path to QEMU binary"
> @echo " QEMU_IMG=/path/to/qemu-img - Change path to qemu-img tool"
> +ifeq ($(PYTHON_YAML),yes)
> + @echo " QEMU_CONFIG=/path/conf.yml - Change path to VM
> configuration .yml file."
> +else
> + @echo " (install python3-yaml to enable support for yaml file to
> configure a VM.)"
> +endif
> + @echo " See conf_example_*.yml for
> file format details."
>
> vm-build-all: $(addprefix vm-build-, $(IMAGES))
>
> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> index 86ec0df24a..0bc1bad839 100644
> --- a/tests/vm/basevm.py
> +++ b/tests/vm/basevm.py
> @@ -478,9 +478,43 @@ class BaseVM(object):
> cwd=cidir,
> stdin=self._devnull, stdout=self._stdout,
> stderr=self._stdout)
> -
> return os.path.join(cidir, "cloud-init.iso")
>
> +def parse_config(config, args):
> + """ Parse yaml config and populate our config structure.
> + The yaml config allows the user to override the
> + defaults for VM parameters. In many cases these
> + defaults can be overridden without rebuilding the VM."""
> + if args.config:
> + config_file = args.config
> + elif 'QEMU_CONFIG' in os.environ:
> + config_file = os.environ['QEMU_CONFIG']
> + else:
> + return config
> + if not os.path.exists(config_file):
> + raise Exception("config file {} does not exist".format(config_file))
> + # We gracefully handle importing the yaml module
> + # since it might not be installed.
> + # If we are here it means the user supplied a .yml file,
> + # so if the yaml module is not installed we will exit with error.
> + try:
> + import yaml
> + except ImportError:
> + print("The python3-yaml package is needed "\
> + "to support config.yaml files")
> + # Instead of raising an exception we exit to avoid
> + # a raft of messy (expected) errors to stdout.
> + exit(1)
> + with open(config_file) as f:
> + yaml_dict = yaml.safe_load(f)
> +
> + if 'qemu-conf' in yaml_dict:
> + config.update(yaml_dict['qemu-conf'])
> + else:
> + raise Exception("config file {} is not valid"\
> + " missing qemu-conf".format(config_file))
> + return config
> +
> def parse_args(vmcls):
>
> def get_default_jobs():
> @@ -517,6 +551,9 @@ def parse_args(vmcls):
> help="run tests with a snapshot")
> parser.add_option("--genisoimage", default="genisoimage",
> help="iso imaging tool")
> + parser.add_option("--config", "-c", default=None,
> + help="Provide config yaml for configuration. "\
> + "See config_example.yaml for example.")
> parser.disable_interspersed_args()
> return parser.parse_args()
>
> @@ -528,6 +565,7 @@ def main(vmcls, config=None):
> if not argv and not args.build_qemu and not args.build_image:
> print("Nothing to do?")
> return 1
> + config = parse_config(config, args)
> logging.basicConfig(level=(logging.DEBUG if args.debug
> else logging.WARN))
> vm = vmcls(args, config=config)
> diff --git a/tests/vm/conf_example_aarch64.yml
> b/tests/vm/conf_example_aarch64.yml
> new file mode 100644
> index 0000000000..9d44ae356f
> --- /dev/null
> +++ b/tests/vm/conf_example_aarch64.yml
> @@ -0,0 +1,51 @@
> +#
> +# Example yaml for use by any of the scripts in tests/vm.
> +# Can be provided as an environment variable QEMU_CONFIG
> +#
> +qemu-conf:
> +
> + # If any of the below are not provided, we will just use the qemu
> defaults.
> +
> + # Login username and password(has to be sudo enabled)
> + guest_user: qemu
> + guest_pass: "qemupass"
> +
> + # Password for root user can be different from guest.
> + root_pass: "qemupass"
> +
> + # If one key is provided, both must be provided.
> + #ssh_key: /complete/path/of/your/keyfile/id_rsa
> + #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub
> +
> + cpu: max
> + machine: virt,gic-version=max
> + memory: 16G
> +
> + # The below is a example for how to configure NUMA topology with
> + # 4 NUMA nodes and 2 different NUMA distances.
> + qemu_args: "-smp cpus=16,sockets=2,cores=8
> + -numa node,cpus=0-3,nodeid=0 -numa node,cpus=4-7,nodeid=1
> + -numa node,cpus=8-11,nodeid=2 -numa node,cpus=12-15,nodeid=3
> + -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15
> + -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20
> + -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20"
> +
> + # By default we do not set the DNS.
> + # You override the defaults by setting the below.
> + #dns: 1.234.567.89
> +
> + # By default we will use a "block" device, but
> + # you can also boot from a "scsi" device.
> + # Just keep in mind your scripts might need to change
> + # As you will have /dev/sda instead of /dev/vda (for block device)
> + boot_dev_type: "block"
> +
> + # By default the ssh port is not fixed.
> + # A fixed ssh port makes it easier for automated tests.
> + #ssh_port: 5555
> +
> + # To install a different set of packages, provide a command to issue
> + #install_cmds: "apt-get update ; apt-get build-dep -y qemu"
> +
> + # Or to skip the install entirely, just provide ""
> + #install_cmds: ""
> diff --git a/tests/vm/conf_example_x86.yml b/tests/vm/conf_example_x86.yml
> new file mode 100644
> index 0000000000..78d3f5830f
> --- /dev/null
> +++ b/tests/vm/conf_example_x86.yml
> @@ -0,0 +1,50 @@
> +#
> +# Example yaml for use by any of the x86 based scripts in tests/vm.
> +# Can be provided as an environment variable QEMU_CONFIG
> +#
> +qemu-conf:
> +
> + # If any of the below are not provided, we will just use the qemu
> defaults.
> +
> + # Login username and password(has to be sudo enabled)
> + guest_user: "qemu"
> + guest_pass: "qemupass"
> +
> + # Password for root user can be different from guest.
> + root_pass: "qemupass"
> +
> + # Provide default ssh keys of current user.
> + # You need to edit the below for your user.
> + #ssh_key_file: /home/<user>/.ssh/id_rsa
> + #ssh_pub_key_file: /home/<user>/.ssh/id_rsa.pub
> +
> + cpu: max
> + machine: pc
> + memory: 8G
> +
> + # The below is a example for how to configure NUMA topology with
> + # 4 NUMA nodes and 2 different NUMA distances.
> + qemu_args: "-smp cpus=8,sockets=2,cores=4
> + -object
> memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node0
> + -object
> memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node1
> + -object
> memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node2
> + -object
> memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node3
> + -numa node,cpus=0-1,nodeid=0 -numa node,cpus=2-3,nodeid=1
> + -numa node,cpus=4-5,nodeid=2 -numa node,cpus=6-7,nodeid=3
> + -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15
> + -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20
> + -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20"
> +
> + # By default we do not set the DNS.
> + # You override the defaults by setting the below.
> + #dns: "1.234.567.89"
> +
> + # By default we will use a "block" device, but
> + # you can also boot from a "scsi" device.
> + # Just keep in mind your scripts might need to change
> + # As you will have /dev/sda instead of /dev/vda (for block device)
> + boot_dev_type: "block"
> +
> + # By default the ssh port is not fixed.
> + # A fixed ssh port makes it easier for automated tests.
> + ssh_port: 5555
--
Alex Bennée
- [PATCH v7 00/12] tests/vm: Add support for aarch64 VMs, Robert Foley, 2020/05/19
- [PATCH v7 01/12] configure: add alternate binary for genisoimage, Robert Foley, 2020/05/19
- [PATCH v7 02/12] tests/vm: pass --genisoimage to basevm script, Robert Foley, 2020/05/19
- [PATCH v7 03/12] tests/vm: pass args through to BaseVM's __init__, Robert Foley, 2020/05/19
- [PATCH v7 04/12] tests/vm: Add configuration to basevm.py, Robert Foley, 2020/05/19
- [PATCH v7 05/12] tests/vm: Added configuration file support, Robert Foley, 2020/05/19
- Re: [PATCH v7 05/12] tests/vm: Added configuration file support,
Alex Bennée <=
- [PATCH v7 06/12] tests/vm: Pass --debug through for vm-boot-ssh., Robert Foley, 2020/05/19
- [PATCH v7 07/12] tests/vm: Add ability to select QEMU from current build., Robert Foley, 2020/05/19
- [PATCH v7 08/12] tests/vm: allow wait_ssh() to specify command, Robert Foley, 2020/05/19
- [PATCH v7 09/12] tests/vm: Added a new script for ubuntu.aarch64., Robert Foley, 2020/05/19
- [PATCH v7 10/12] tests/vm: Added a new script for centos.aarch64., Robert Foley, 2020/05/19