Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console

[Philippe Mathieu-Daudé]

Cross-posting qemu-devel + avocado-devel.

> While previously working on a Super I/O refactor, I encountered some problems
> at runtime, after building the codebase successfully and running qtests.
> I had to manually start to boot different guests and check the bootlog.
> 
> I wanted to give a try at Avocado which seems designed to simplify that kind
> of functional tests.
> 
> I applied Amador Pahim work following Cleber Rosa documentation from
> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
> however I had to modify few things to parse the boot console.
> Since his work is not merged, I included it in this series.
> 
> The tests simply expect to find a string reported by Linux printk when a
> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 
> 16550A"
> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
> device are correctly detected:
> 
> 
>     self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>     self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>     self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>     self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
> 
> Example of the tests output:
> 
>     $ avocado run test_linux-boot-console.py -m 
> test_linux-boot-console.py.data/parameters.yaml 
>     JOB ID     : 695094c9bbe8f6011226da7c2031c2c53e949910
>     JOB LOG    : 
> /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>      (1/6) 
> test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582:
>  PASS (4.76 s)
>      (2/6) 
> test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72:
>  PASS (0.00 s)
>      (3/6) 
> test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582:
>  PASS (0.00 s)
>      (4/6) 
> test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72:
>  PASS (3.92 s)
>      (5/6) 
> test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582:
>  PASS (0.00 s)
>      (6/6) 
> test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72:
>  PASS (4.08 s)
>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>     JOB TIME   : 13.31 s
> 
> Please apologize my ugly Python, this series is just a proof-of-concept :)
> I couldn't figure out how to use the @skipUnless(correct arch) decorator.

Eduardo asked me to share my first impressions after trying this
framework with QEMU.

So far it seems more designed to run tests _inside_ a qemu instance
(think user-space), eventually testing packages/scripts.

My tests are targeting the machine model itself, if the devices are
correctly instantiated and so.
The fastest approach was to check the Linux kernel bootlog, but if you
look at the TestAlphaClipperBoot console you'll see a boot "firmware" is
executed before the kernel. Casually this firmware also send information
on the console, but what if the console is not accessible?
We can use a chardev for the ioport80 POST, but some firmwares post boot
events via I2C, SPI, CAN...

To improve the testing, my idea is to use the Tracing framework.
The machine trace events would be logged in a db and avogado would
verify some of the trace events, did they occur? in the correct sequence
order? with the correct arguments? We can also check relative
timestamps, is this sequence timing fast enough?

I plan to use the gdb API to insert breakpoints and follow part
execution flow, eventually injecting (faulty) events.

The multi-arch/machines support is a bit weak yet.

I'm personally more interested in automatically testing [real world]
firmware or full machine boot process, the kind of integration testing I
can not do with qtests.

I liked the possibility to generate coredumps, or the replay function.

I also liked to be able to write a test on how a machine boots, in ~20
LOC (see TestAlphaClipperBoot2_6).

The storage API is OK to fetch a full VM image, but not a single file
like a kernel or a flash image.

Enough for my second try, good work :)

I hope this was helpful.

Regards,

Phil.

signature.asc
Description: OpenPGP digital signature