Re: Collecting GRUB 2.06 test results

[Glenn Washburn]

On Wed, 10 Feb 2021 20:29:41 +0100
John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote:

> Hello Glenn!
> 
> On 2/10/21 8:21 PM, Glenn Washburn wrote:
> > Thanks for this these tests. Are you running these tests on the
> > same architecture that you're testing? Or, for instance, running the
> > tests for all architectures on say x86_64?
> 
> Those were all tested on native hardware with a few exceptions:
> 
> - 32-bit PowerPC tests were run on a 64-bit PowerPC machine
> - 32-bit MIPS tests were run on a 64-bit MIPS machine
> 
> I do have an iBook G4 and could run the 32-bit PowerPC tests there.
> 
> (Now I'm actually not sure anymore whether I ran the 32-bit PowerPC
>  tests on the G4, but I can still do that. The machine is next to
>  me, just currently powered off).

Ok, I'm glad you're running the tests on native hardware and in that
case I'm not surprised most of the qemu tests aren't running because I
assume that qemu is not on most of the non x86 platforms. Perhaps we
should have the tests show as SKIPPED when the appropriate qemu binary
can not be found.

> > I'd like to note that it appears that these tests do not perform
> > most of the various filesystem tests.
> 
> I ran the tests with "make check" (or "make test"), I did not make
> any modifications. If some tests were not run that's because GRUB does
> not enable them on the target in question.

That's technically true, but not the whole story. GRUB is not enabling
the filesystem tests because you don't have your target environment
setup such that the tests will be enabled. The filesystem tests require
filesystem tools (eg. e2fsprogs for ext* fs tests). You might try
installing the prerequisites needed by the tests, and then seeing if
they run.

> > Also, I believe mips arch is not being tested here. And it appears
> > that none of the qemu tests for mipsle are being run due to not
> > having qemu for that arch installed.
> 
> What do you mean by "here"?

I mean that it appears to me that you don't have tests for the mips
arch, only mipsle. I'm not very familiar with mips, so I may be
confusing. My understanding is that there are big and little endian
variants of the mips architecture, and 32 vs 64 bit variants. I
understand mipsle to indicate the little-endian variant, but GRUB
supports the big endian variant as well. It seems to me that you only
have test logs for the 32 and 64 bit little endian variants.

> > None of the qemu tests are being run for most of the architectures
> > (I also haven't figured out how to get them to run on most
> > architectures). But I have gotten the qemu tests to run for arm and
> > arm64, which they are not running in your tests. What is especially
> > odd, is that for armhf grub-shell is looking for 'qemu-system-i386'
> > and not finding it. But it should be looking for the arm qemu
> > variant.
> > 
> > For completeness, The risc architecture is not being tested either.
> > Yes, I know most tests fail because of a bug. Are we considering it
> > an unsupported platform because its unusable, even though it can be
> > built?
> 
> I currently do not have access to a RISC-V machine yet. I have
> preordered one and also applied for a developer machine from SiFive,
> but I haven't gotten any yet.
>
> > Where is the list of architectures "supported" by GRUB? Or is it all
> > the ones allowed to be built by master?
> 
> You can see that by looking at the configure.ac. It's basically all
> targets that build more than just the utilities.

Great, that's what I was thinking. I notice that it appears you're not
testing every supported target, just one target from each architecture.
For instance, you're not testing and i386 build of the ieee1275 or efi
platforms. 

Since you're not using qemu on the non-x86 platforms, I'm not sure if
testing the various platforms of other architectures would be worth
while (eg. arm-coreboot and arm-efi). Perhaps someone more
knowledgeable than I can weight in.

Glenn