Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Eric Auger]

Hi William,

On 6/29/21 10:38 PM, Willian Rampazzo wrote:
> On Tue, Jun 29, 2021 at 5:17 PM Eric Auger <eric.auger@redhat.com> wrote:
>> Hi Cleber, all,
>>
>> On 6/29/21 4:36 PM, Eric Auger wrote:
>>> This series adds ARM SMMU and Intel IOMMU functional
>>> tests using Fedora cloud-init images.
>>>
>>> ARM SMMU tests feature guests with and without RIL
>>> (range invalidation support) using respectively fedora 33
>>> and 31.  For each, we test the protection of virtio-net-pci
>>> and virtio-block-pci devices. Also strict=no and passthrough
>>> modes are tested. So there is a total of 6 tests.
>>>
>>> The series applies on top of Cleber's series:
>>> - [PATCH 0/3] Acceptance Tests: support choosing specific
>>>
>>> Note:
>>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
>>> test_smmu_noril_nostrict) pass but the log reports:
>>> "WARN: Test passed but there were warnings during execution."
>>> This seems due to the lack of hash when fetching the kernel and
>>> initrd through fetch_asset():
>>> WARNI| No hash provided. Cannot check the asset file integrity.
>> I wanted to emphasize that point and wondered how we could fix that
>> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
>> Any advice?
> Hi Eric,
>
> We had that discussion some weeks ago regarding the WARN status of a
> test when the file hash is not provided for the fetch call. We agreed
> that a WARN is not a harmful status, and it would be okay.
>
> When we got the request to add the message regarding a missing hash of
> a downloaded file, we concluded that it would not make sense to set it
> like a normal message in the logs because no one would open the logs
> and see the message if the test succeed.
>
> If you think a WARN may be considered a harmful status, let us know,
> and we can try to adjust it so that users see the message when a hash
> is not provided without setting the test status as WARN.
>
> On the other hand, you can always add the hash if you have access to it.
>
> I hope it helps somehow,
It does, thank you for your reply.

Maybe we should tag the tests with warnings as PASS in the last summary
RESULT line and keep the individual test result lines as they are.

Currently it is not obvious the tests has passed if you look at the last
line. See my logs below.



 (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (106.83 s)
 (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough: WARN:
Test passed but there were warnings during execution. Check the log for
details. (112.23 s)
 (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
Test passed but there were warnings during execution. Check the log for
details. (110.77 s)
 (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (135.18 s)
 (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
Test passed but there were warnings during execution. Check the log for
details. (115.98 s)
 (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN: Test
passed but there were warnings during execution. Check the log for
details. (137.16 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0 |
CANCEL 0

Thanks

Eric

>
> Willian
>
>> Best Regards
>>
>> Eric
>>> History:
>>> v3 -> v4:
>>> - I added Wainer's refactoring of KNOWN_DISTROS
>>> into a class (last patch) and took into account his comments.
>>>
>>> v2 -> v3:
>>> - Added Intel IOMMU tests were added. Different
>>> operating modes are tested such as strict, caching mode, pt.
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> The series and its dependencies can be found at:
>>> https://github.com/eauger/qemu/tree/avocado-qemu-v4
>>>
>>> Eric Auger (3):
>>>   Acceptance Tests: Add default kernel params and pxeboot url to the
>>>     KNOWN_DISTROS collection
>>>   avocado_qemu: Add SMMUv3 tests
>>>   avocado_qemu: Add Intel iommu tests
>>>
>>> Wainer dos Santos Moschetta (1):
>>>   avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class
>>>
>>>  tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
>>>  tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
>>>  tests/acceptance/smmu.py                  | 132 ++++++++++++++++++++++
>>>  3 files changed, 332 insertions(+), 33 deletions(-)
>>>  create mode 100644 tests/acceptance/intel_iommu.py
>>>  create mode 100644 tests/acceptance/smmu.py
>>>