Re: [Qemu-devel] make check speed

[Lukáš Doktor]

Dne 23.8.2017 v 10:35 Thomas Huth napsal(a):
> On 23.08.2017 10:01, Paolo Bonzini wrote:
>> On 23/08/2017 09:49, Thomas Huth wrote:
>>> While we're at it: I'd like to have a "make check-fast", too. Sometimes
>>> the normal "make check" is already too slow, e.g. while developing new
>>> patches, I sometimes just want to do a very quick sanity test to see
>>> whether I broke some basic things or not, and only do the "make check"
>>> before I submit my patches.
>>> So we would get three stages:
>>>
>>> - make check-fast => For very, very quick sanity tests only
>>>
>>> - make check => E.g. has to be run before submitting patches
>>>
>>> - make check-harder => might run a very long time, so best suited for
>>>                        nightly regression tests etc.?
>>>
>>> Does that sound reasonable? And the crucial question: Who is going to
>>> implement the basic framework for this?
>>
>> There's already make check-unit or make check-qtest-x86_64 depending on
>> what you're working on.
> 
> True. And I just learned that you can also already set the SPEED
> variable to either "quick" or "slow" and that we're already using
> g_test_quick() and g_test_slow() in a couple of places to check this. So
> the framework for running quick vs. thorough tests is already there ...
> we just might want to add this to some more tests, I guess...
> 
> Question for the maintainers and the test automation folks: Is anybody
> already running "make check SPEED=slow" or is this just rather an
> unheard-of way of running the tests?
> 
>> If you have a many-core machine, of course, there's no simpler solution
>> than throwing more CPUs at it. :)
> 
> Is it safe nowadays to run "make check -j4" for example? Last time I
> tried (maybe 1 or 2 years ago), there were still issues since some tests
> were using hard-coded temporary file names, so the parallel tests were
> disturbing each other, for example...
> 

Actually the `.travis.yml` defines `MAKEFLAGS="-j3"`, which results in `make 
check` being executed with 3 threads...

I was actually looking at the increasing number of failed travis builds and it 
seems to be related to the fluctuating performance. Running `nice -n 20 make 
check -j 12` with `nice -n 5 stress -c 20` in background results in the same 
kind of failures:

    File '/tmp/qemu/include/qapi/qmp/qobject.h'
    Lines executed:0.00% of 9
    **
    ERROR:tests/prom-env-test.c:42:check_guest_memory: assertion failed 
(signature == MAGIC): (0x7c7f1b78 == 0xcafec0de)
    GTester: last random seed: R02S117a2b838f1fd17c65a134629577b996
    make: *** [check-qtest-ppc] Chyba 1
    /tmp/qemu/tests/Makefile.include:836: návod pro cíl „check-qtest-ppc“ selhal
    make: *** [check-qtest-sparc] Chyba 1
    /tmp/qemu/tests/Makefile.include:836: návod pro cíl „check-qtest-sparc“ 
selhal
    **
    ERROR:tests/rtc-test.c:173:check_time: assertion failed (ABS(t - s) <= 
wiggle): (3 <= 2)
    GTester: last random seed: R02S2ca82eb2e7f63ec62c8433b715d9fe12
    Vhost user backend fails to broadcast fake RARP
    **
    ERROR:tests/vhost-user-test.c:779:test_reconnect: child process 
(/i386/vhost-user/reconnect/subprocess [5264]) failed unexpectedly
    GTester: last random seed: R02S6e728ccb5384a4d856cacc4166be8052
    Gcov report for hw/misc/tmp105.c:
    File 'hw/misc/tmp105.c'
    Lines executed:30.40% of 125
    Gcov report for arm-softmmu/hw/block/virtio-blk.c:
    File '/tmp/qemu/hw/block/virtio-blk.c'

and all ERROR lines:

    ERROR:tests/prom-env-test.c:42:check_guest_memory: assertion failed 
(signature == MAGIC): (0x00000000 == 0xcafec0de)
    ERROR:tests/rtc-test.c:173:check_time: assertion failed (ABS(t - s) <= 
wiggle): (3 <= 2)
    ERROR:tests/vhost-user-test.c:779:test_reconnect: child process 
(/i386/vhost-user/reconnect/subprocess [5264]) failed unexpectedly
    ERROR:tests/vhost-user-test.c:807:test_connect_fail: child process 
(/x86_64/vhost-user/connect-fail/subprocess [7751]) failed unexpectedly
    ERROR:tests/vhost-user-test.c:835:test_flags_mismatch: child process 
(/x86_64/vhost-user/flags-mismatch/subprocess [7892]) failed unexpectedly
    ERROR:tests/boot-sector.c:127:boot_sector_test: assertion failed (signature 
== SIGNATURE): (0x00000000 == 0x0000dead)

on my machine. Using `make -j 1` or at least `make -j 2` could improve (but I 
haven't checked that)

Lukáš

>  Thomas
>

signature.asc
Description: OpenPGP digital signature