Re: [Qemu-devel] [RFC 17/18] validator.py script

[Markus Armbruster]

Eduardo Habkost <address@hidden> writes:

> See cover letter for a description of the new test system.
>
> TODO: code clean up
> TODO: write description.
>
> Signed-off-by: Eduardo Habkost <address@hidden>
> ---
>  scripts/validator.py | 724 
> +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 724 insertions(+)
>  create mode 100755 scripts/validator.py
>
> diff --git a/scripts/validator.py b/scripts/validator.py
> new file mode 100755
> index 0000000000..4312571feb
> --- /dev/null
> +++ b/scripts/validator.py
> @@ -0,0 +1,724 @@
> +#!/usr/bin/env python
> +#
> +#  Copyright (c) 2018 Red Hat Inc
> +#
> +# Author:
> +#  Eduardo Habkost <address@hidden>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +"""
> +QEMU validator script
> +=====================
> +
> +This script will get test YAML test case specifications or Python
> +modules as input, and generate/run test cases based on them.
> +
> +USAGE
> +-----
> +
> +validator.py <specification-file>... -V VAR1=value1 VAR1=value2 VAR2=value3
> +
> +specification-file is a YAML file containing the test specification.

I can see the "test YAML test case specifications", but not the "Python
modules".

Should we introduce yet another markup language into QEMU?  (pardon the
pun).

> +
> +Example::
> +
> +    # this test specification is equivalent to the
> +    # "device/introspect/list" test case in device-introspect-test.c

If we merge this, not for long :)

> +    command-line: '$QEMU -nodefaults -machine none'
> +    monitor-commands:
> +    - qmp:
> +      - execute: qom-list-types
> +        arguments:
> +          implements: 'device'
> +          abstract: true
> +    - hmp: 'device_add help'
> +
> +
> +VARIABLE EXPANSION
> +------------------
> +
> +The test runner will try to run the test cases with all possible values
> +for variables appearing in the test specification.
> +
> +Some built-in variables are automatically expanded:
> +
> +* `$MACHINE` - Expands to a machine-type name supported by $QEMU
> +* `$ACCEL` - Expands to an accelerator name supported by $QEMU
> +* `$DEVICE` - Expands to a (user-creatable) device type name supported by 
> $QEMU
> +* `$CPU` - Expands to a CPU model name supported by $QEMU
> +
> +Note that the $QEMU variable must be specified in th

Yes?

> +
> +TEST SPECIFICATION FIELDS
> +-------------------------
> +
> +command-line
> +~~~~~~~~~~~~
> +
> +List or string, containing the QEMU command-line to be run.
> +
> +Default: '$QEMU'
> +
> +
> +monitor-commands
> +~~~~~~~~~~~~~~~~
> +
> +Mapping or list-of-mappings containing monitor commands to run.  The key on 
> each
> +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
> +mapping, or list.
> +
> +Default: None.
> +
> +TODO: not implemented yet.

The whole monitor-commands feature?

Can I do

    monitor-commands:
    - qmp:
      - execute: eins
    - hmp: zwei
    - qmp:
      - execute: drei

to execute eins, zwei, drei in this order?

> +
> +
> +qmp
> +~~~
> +
> +Boolean.  If true (the default), a QMP monitor is configured on the 
> command-line
> +automatically.
> +
> +If true, the test runner will issue a ``quit`` command automatically when
> +testing is finished.  If false, the test runner will wait until QEMU exits by
> +itself.
> +
> +Example::
> +
> +    # just run $QEMU -help and ensure it won't crash
> +    command-line: ['$QEMU', '-help']
> +    qmp: false
> +
> +
> +TODO: whitelist
> +TODO: validate output against reference output
> +TODO: configure defaults for variables
> +TODO: compatibility with Avocado multiplexer?
> +"""

This is a DSL to write tests.  I applaud the idea to write more tests at
a higher level than C.  We just need to be careful not to create too
many different ways to write tests, or else readability will suffer.
Ideally, the way to use for a test should be fairly obvious.

Looking forward to your next iteration.

[...]