Re: [PATCH v3 0/9] tests: Add test cases for TPM 1.2 ACPI tables

[Markus Armbruster]

Igor Mammedov <imammedo@redhat.com> writes:

> On Mon, 19 Jul 2021 15:56:16 +0200
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>
>> On Mon, Jul 19, 2021 at 3:50 PM Markus Armbruster <armbru@redhat.com> wrote:
>> > Igor Mammedov <imammedo@redhat.com> writes:  
>> > > On Thu, 15 Jul 2021 07:49:13 +0200
>> > > Markus Armbruster <armbru@redhat.com> wrote:  
>> > >> Philippe Mathieu-Daudé <philmd@redhat.com> writes:  
>> 
>> > >> >>> IMO the "right" solution is to check via QMP if TMP is supported
>> > >> >>> or not. This is now doable since commit caff255a546 ("tpm: Return
>> > >> >>> QMP error when TPM is disabled in build").
>> > >> >>>
>> > >> >>> Long term we'd like to decouple the tests/ build from the various
>> > >> >>> QEMU configurations, and build the tests once.  
>> > >> >>
>> > >> >> This argument applies only to macros from target-specific headers 
>> > >> >> like
>> > >> >> $TARGET-config-target.h, not to macros from config-host.h.  #ifdef
>> > >> >> CONFIG_TPM should be fine, shouldn't it?  
>> > >> >
>> > >> > Some definitions depend on the host (OS, libraries installed, ...),
>> > >> > others depend on the --enable/--disable ./configure options.
>> > >> >
>> > >> > IMO it would be nice if we could get qtests independent of the 
>> > >> > latter.  
>> > >>
>> > >> Why?  
>> > >
>> > > In another mail-thread Philippe mentioned that there is desire
>> > > to use qtest out of tree to test other QEMU binaries.
>> > >
>> > > However, just probing for features at runtime aren't going
>> > > to help with the goal as tests are tailored for the latest
>> > > CLI/QMP/ABI. To make it work we would have practically
>> > > introduce versioned tests.
>> > >
>> > > So I wonder why one external acceptance-tests suite is not
>> > > sufficient, that we would want to hijack relatively simple
>> > > internal qtest at expense of increased resources needed to
>> > > run/write unit tests.  
>> >
>> > Yes.  qtest was not designed for use with anything but HEAD, and I doubt
>> > we can make it fit such uses at reasonable expense.  
>> 
>> One HEAD but multiple configurations...
>
> Even assuming reconfigure won't cause world rebuild,
> It will be a win only if number of configuration probes
> is small.
> However it doesn't scale for large numbers and it might be
> faster to rebuild affected tests in the end. (worst case: #probes * #targets)
> I wonder if we can do probing once & cache it somewhere to avoid ^^^.

As far as I can tell, the time to compile these tests is dwarved several
times over by the time to run them.  In other words, most of the waste
to optimize out hides in the test programs, not in compiling them.

>> If you want to simplify human time, can we simply run qtests once per
>> arch/OS but with all features enabled? Otherwise skip qtests?

How build configuration and tests interact is poorly understood.  We
instead use brute force: run all the tests for all configurations,
always.  To do better without sacrificing coverage, we need to
understand better.  Sadly, I don't see that happening.