Re: Failing avocado tests in CI (was: Re: [PULL 00/24] tcg + linux-user queue for 8.1-rc3)

[Richard Henderson]

On 8/23/23 06:04, Thomas Huth wrote:
> On 06/08/2023 05.36, Richard Henderson wrote:
> > The following changes since commit 6db03ccc7f4ca33c99debaac290066f4500a2dfb:
> >
> >    Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging (2023-08-04 
> > 14:47:00 -0700)
> >
> > are available in the Git repository at:
> >
> >    https://gitlab.com/rth7680/qemu.git tags/pull-tcg-20230805
> >
> > for you to fetch changes up to 843246699425adfb6b81f927c16c9c6249b51e1d:
> >
> >    linux-user/elfload: Set V in ELF_HWCAP for RISC-V (2023-08-05 18:17:20 +0000)
> >
> > ----------------------------------------------------------------
> > accel/tcg: Do not issue misaligned i/o
> > accel/tcg: Call save_iotlb_data from io_readx
> > gdbstub: use 0 ("any process") on packets with no PID
> > linux-user: Fixes for MAP_FIXED_NOREPLACE
> > linux-user: Fixes for brk
> > linux-user: Adjust task_unmapped_base for reserved_va
> > linux-user: Use ELF_ET_DYN_BASE for ET_DYN with interpreter
> > linux-user: Remove host != guest page size workarounds in brk and image load
> > linux-user: Set V in ELF_HWCAP for RISC-V
> > *-user: Remove last_brk as unused
>
>   Hi Richard,
>
> I noticed that we currently have two failing Avocado jobs in our CI, avocado-system-centos 
> and avocado-system-opensuse, where the boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg and 
> the boot_linux.py:BootLinuxX8664.test_pc_q35_tcg are now apparently crashing. If I've got 
> the history right, it started with your pull request here, in the preceeding one from 
> Paolo, everything is still green:
>
>   https://gitlab.com/qemu-project/qemu/-/pipelines/956543770
>
> But here the jobs started failing:
>
>   https://gitlab.com/qemu-project/qemu/-/pipelines/957458385
>
> Could you please have a look?

It's some sort of timing issue, which sometimes goes away when re-run.  I was re-running 
tests *a lot* in order to get them to go green while running the 8.1 release.

For instance, with very little added except for your s390x pull, the same 
BootLinuxX8664.test_pc_i440fx_tcg test passes:

https://gitlab.com/qemu-project/qemu/-/jobs/4931341744#L136

In the failing i44fx_tcg test, you can even see it's a timing issue:

https://qemu-project.gitlab.io/-/qemu/-/jobs/4813804725/artifacts/build/tests/results/latest/test-results/02-tests_avocado_boot_linux.py_BootLinuxX8664.test_pc_i440fx_tcg/debug.log

23:42:30 DEBUG| [   61.003328] Sending NMI from CPU 0 to CPUs 1:
23:42:30 DEBUG| [   61.007829] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long 
to run: 2.622 msecs
23:42:30 DEBUG| [   61.003328] NMI backtrace for cpu 1 skipped: idling at 
native_safe_halt+0xe/0x10
23:42:30 DEBUG| [   61.003328] rcu: rcu_sched kthread starved for 60002 jiffies! g-963 
f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
23:42:30 DEBUG| [   61.003328] rcu: RCU grace-period kthread stack dump:
23:42:30 DEBUG| [   61.003328] rcu_sched       I    0    10      2 0x80004000
23:42:30 DEBUG| [   61.003328] Call Trace:
23:42:30 DEBUG| [   61.003328]  ? __schedule+0x29f/0x680
...


r~