Re: [PATCH 2/2] docs: add definitions of terms for CI/testing

[Philippe Mathieu-Daudé]

On 8/20/21 11:09 PM, Willian Rampazzo wrote:
> To understand the current state of QEMU CI/testing and have a base to
> discuss the plans for the future, it is important to define some usual
> terms. This patch defines the terms for "Automated tests", "Unit
> testing", "Functional testing", "System testing", "Flaky tests",
> "Gating", and "Continuous Integration".
> 
> The first patch was borrowed from
> 20210812180403.4129067-1-berrange@redhat.com.
> 
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> ---
>  docs/devel/ci-definitions.rst | 121 ++++++++++++++++++++++++++++++++++
>  docs/devel/ci.rst             |   1 +
>  2 files changed, 122 insertions(+)
>  create mode 100644 docs/devel/ci-definitions.rst

Wow, super cool!

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> diff --git a/docs/devel/ci-definitions.rst b/docs/devel/ci-definitions.rst
> new file mode 100644
> index 0000000000..32e22ff468
> --- /dev/null
> +++ b/docs/devel/ci-definitions.rst
> @@ -0,0 +1,121 @@
> +Definition of terms
> +===================
> +
> +This section defines the terms used in this document and correlates them with
> +what is currently used on QEMU.
> +
> +Automated tests
> +---------------
> +
> +An automated test is written on a test framework using its generic test
> +functions/classes. The test framework can run the tests and report their
> +success or failure [1]_.
> +
> +An automated test has essentially three parts:
> +
> +1. The test initialization of the parameters, where the expected parameters,
> +   like inputs and expected results, are set up;
> +2. The call to the code that should be tested;
> +3. An assertion, comparing the result from the previous call with the 
> expected
> +   result set during the initialization of the parameters. If the result
> +   matches the expected result, the test has been successful; otherwise, it 
> has
> +   failed.
> +
> +Unit testing
> +------------
> +
> +A unit test is responsible for exercising individual software components as a
> +unit, like interfaces, data structures, and functionality, uncovering errors
> +within the boundaries of a component. The verification effort is in the
> +smallest software unit and focuses on the internal processing logic and data
> +structures. A test case of unit tests should be designed to uncover errors 
> due
> +to erroneous computations, incorrect comparisons, or improper control flow 
> [2]_.
> +
> +On QEMU, unit testing is represented by the 'check-unit' target from 'make'.
> +
> +Functional testing
> +------------------
> +
> +A functional test focuses on the functional requirement of the software.
> +Deriving sets of input conditions, the functional tests should fully exercise
> +all the functional requirements for a program. Functional testing is
> +complementary to other testing techniques, attempting to find errors like
> +incorrect or missing functions, interface errors, behavior errors, and
> +initialization and termination errors [3]_.
> +
> +On QEMU, functional testing is represented by the 'check-qtest' target from
> +'make'.
> +
> +System testing
> +--------------
> +
> +System tests ensure all application elements mesh properly while the overall
> +functionality and performance are achieved [4]_. Some or all system 
> components
> +are integrated to create a complete system to be tested as a whole. System
> +testing ensures that components are compatible, interact correctly, and
> +transfer the right data at the right time across their interfaces. As system
> +testing focuses on interactions, use case-based testing is a practical 
> approach
> +to system testing [5]_. Note that, in some cases, system testing may require
> +interaction with third-party software, like operating system images, 
> databases,
> +networks, and so on.
> +
> +On QEMU, system testing is represented by the 'check-acceptance' target from
> +'make'.
> +
> +Flaky tests
> +-----------
> +
> +A flaky test is defined as a test that exhibits both a passing and a failing
> +result with the same code on different runs. Some usual reasons for an
> +intermittent/flaky test are async wait, concurrency, and test order 
> dependency
> +[6]_.
> +
> +Gating
> +------
> +
> +A gate restricts the move of code from one stage to another on a
> +test/deployment pipeline. The step move is granted with approval. The 
> approval
> +can be a manual intervention or a set of tests succeeding [7]_.
> +
> +On QEMU, the gating process happens during the pull request. The approval is
> +done by the project leader running its own set of tests. The pull request 
> gets
> +merged when the tests succeed.
> +
> +Continuous Integration (CI)
> +---------------------------
> +
> +Continuous integration (CI) requires the builds of the entire application and
> +the execution of a comprehensive set of automated tests every time there is a
> +need to commit any set of changes [8]_. The automated tests can be composed 
> of
> +the unit, functional, system, and other tests.
> +
> +Keynotes about continuous integration (CI) [9]_:
> +
> +1. System tests may depend on external software (operating system images,
> +   firmware, database, network).
> +2. It may take a long time to build and test. It may be impractical to build
> +   the system being developed several times per day.
> +3. If the development platform is different from the target platform, it may
> +   not be possible to run system tests in the developer’s private workspace.
> +   There may be differences in hardware, operating system, or installed
> +   software. Therefore, more time is required for testing the system.
> +
> +References
> +----------
> +
> +.. [1] Sommerville, Ian (2016). Software Engineering. p. 233.
> +.. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
> +       A Practitioner’s Approach. p. 48, 376, 378, 381.
> +.. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
> +       A Practitioner’s Approach. p. 388.
> +.. [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
> +       A Practitioner’s Approach. Software Engineering, p. 377.
> +.. [5] Sommerville, Ian (2016). Software Engineering. p. 59, 232, 240.
> +.. [6] Luo, Qingzhou, et al. An empirical analysis of flaky tests.
> +       Proceedings of the 22nd ACM SIGSOFT International Symposium on
> +       Foundations of Software Engineering. 2014.
> +.. [7] Humble, Jez & Farley, David (2010). Continuous Delivery:
> +       Reliable Software Releases Through Build, Test, and Deployment, p. 
> 122.
> +.. [8] Humble, Jez & Farley, David (2010). Continuous Delivery:
> +       Reliable Software Releases Through Build, Test, and Deployment, p. 55.
> +.. [9] Sommerville, Ian (2016). Software Engineering. p. 743.
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index a6a650968b..8d95247188 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -8,5 +8,6 @@ found at::
>  
>     https://wiki.qemu.org/Testing/CI
>  
> +.. include:: ci-definitions.rst
>  .. include:: ci-jobs.rst
>  .. include:: ci-runners.rst
>