Should we keep using Avocado for functional testing? (was: Re: [PULL 22/23] hw/sd: Fix incorrect populated function switch status data structure)

[Philippe Mathieu-Daudé]

On 10/23/20 11:23 AM, Philippe Mathieu-Daudé wrote:
> On 10/23/20 4:02 AM, Bin Meng wrote:
> > Hi Niek,
> >
> > On Thu, Oct 22, 2020 at 11:20 PM Niek Linnenbank
> > <nieklinnenbank@gmail.com> wrote:
> > > Hi Bin, Philippe,
> > >
> > > If im correct the acceptance tests for orange pi need to be run with 
> > > a flag ARMBIAN_ARTIFACTS_CACHED set that explicitly allows them to be 
> > > run using the armbian mirror. So if you pass that flag on the same 
> > > command that Philippe gave, the rests should run.
> >
> > Thank you for the hints. Actually I noticed the environment variable
> > ARMBIAN_ARTIFACTS_CACHED when looking at the test codes, but after I
> > turned on the flag it still could not download the test asset from the
> > apt.armbian.com website.
>
> This patch could help you, but it use a different image [*]:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg737760.html
> (I haven't tested the image yet but I assume the same bug is present).
>
> > > I have a follow up question and Im interested to hear your opinion on 
> > > that Philippe. Should we perhaps update the orange pi tests (and 
> > > maybe others) so they use a reliable mirror that we can control, for 
> > > example a github repo? I would be happy to create a repo for that, at 
> > > least for the orange pi tests. But maybe there is already something 
> > > planned as a more general solution for artifacts of other machines as 
> > > well?
>
> This is a technical debt between the Avocado and QEMU communities
> (or a misunderstanding?).
>
> QEMU community understood the Avocado caching would be an helpful
> feature, but it ended being more of a problem.
>
>
> I.e. here Niek, Michael Roth and myself still have the image cached,
> so we can keep running the tests committed to the repository. You
> Bin can not, as the armbian mirror is not stable and remove the old
> image.
>
> The old image (Armbian_19.11.3_Orangepipc_bionic_current_5.3.9.img)
> has been manually tested by Niek and myself, I enabled tracing and
> look at the SD packets for some time, was happy how the model worked
> and decided we should keep running a test with this particular image.
>
> Armbian released new images, in particular the one Cleber sent in [*]:
> Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz. While it might
> work similarly, I haven't tested it, and don't have time to do again
> the same audit.
>  From my experience with the Raspberry Pi, these images never work the
> same way, as the Linux kernel evolves quickly with these kinda recent
> embedded boards. The QEMU raspi models have to emulate new devices
> (or complete current ones) every years, because Linux started to use
> a device differently, or started to use a part of the SoC that was not
> used before. Either the kernel doesn't boot, or there are side-effect
> later when userspace program is executed. PITA to debug TBH. For this
> reason I disagree with updating test images to the latest releases.
>
> It would be nice to know QEMU works with the latest Armbian, but
> nobody popped up on the mailing list asking for it. I am personally
> happy enough using the 19.11 release for now.

Back to the cache problem, I started to ask 2 years ago,
https://www.mail-archive.com/avocado-devel@redhat.com/msg00860.html
note this is the exact situation we are having:

>> - What will happens if I add tests downloading running on their
>> compiled u-boot
>> 
(https://downloads.gumstix.com/images/angstrom/developer/2012-01-22-1750/u-boot.bin)
>> and the company decides to remove this old directory?
>> Since sometimes old open-source software are hard to rebuild with
>> recent compilers, should we consider to use a public storage to keep
>> open-source (signed) blobs we can use for integration testing?
>
> For Avocado-VT, there are the JeOS images[1], which we keep on a test
> "assets" directory.  We have a lot of storage/bandwidth availability,
> so it can be used for other assets proven to be necessary for tests.
>
> As long as distribution rights and licensing are not issues, we can
> definitely use the same server for kernels, u-boot images and what
> not.
>
> [1] - https://avocado-project.org/data/assets/


We discussed about this again last year at the KVM forum. Various
RFE have been filled:
https://www.mail-archive.com/avocado-devel@redhat.com/msg01183.html


What we need is a QEMU community file server similar to the asset
one used by the Avocado community. The problem is who is going to
pay and sysadmin it. IIRC Gerd suggested to use GitLab Page, but
the artifact file size is limited to a maximum of 1GiB:
https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-pages
Alternative is to use a git-lfs server:
https://docs.gitlab.com/ee/topics/git/lfs/#hosting-lfs-objects-externally
This is now a project management problem.

Next week is the KVM forum 2020 and there might be a BoF session
to talk about that:
https://kvmforum2020.sched.com/overview/type/Birds+of+a+Feather+Sessions+(BoF)


Now that the QEMU community started to use gitlab-ci more and more
I received complains that Avocado is not practical (hard to
understand how to reproduce command line to debug, options to use
not clear, timeouts not clear, why download all artifacts to run
a single test, various issues regarding caching, cache filled /home
filesystem) so I have been asked to look at Avocado alternatives,
because we want contributors run more tests and CI, not them be
scared by it.
I haven't looked at alternatives yet, from the various features
we could have use, the biggest one is the ability to interact with
the serial console. And that feature is duplicated with the VM
testing class: tests/vm/basevm.py.
The SSH session used in linux_ssh_mips_malta.py is similar to the
one from basevm.py too (see 'make vm-boot-ssh-%').
Some tests are not 'acceptance' but simple qtest written in Python,
such cpu_queries.py / pc_cpu_hotplug_props.py / migration.py /
pc_cpu_hotplug_props.py / x86_cpu_model_versions.py.

The classes I find practical are downloading artifact, uncompressing
or extracting archive, and eventually the cloudinit one. But we can 
probably use them directly.


Lot of material to discuss :)

Regards,

Phil.