Re: [PULL 00/11] QEMU changes for 2021-03-02

[Daniel P . Berrangé]

On Fri, Mar 04, 2022 at 06:46:51PM +0000, Peter Maydell wrote:
> On Fri, 4 Mar 2022 at 17:41, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > The test seems to be flaky, I've been fighting with it all week---trying
> > multiple versions of this pull request and removing patches until
> > build-oss-fuzz passed.  The set of patches that triggered it or not was
> > completely random, but I'll not that it did pass with this exact commit
> > I'm submitting (https://gitlab.com/bonzini/qemu/-/jobs/2154365356).
> >
> > I wanted to look at this today again before replying to you, but as you
> > know I was sidetracked by work on the qemu.org infrastructure.  So, I
> > can look at this but I really need to ask you one of two favors:
> >
> > 1) decide that the test is flaky and merge this pull request, and then
> > I'll send before Monday the changes that I've omitted here (which again
> > have nothing to do with qos-test).  I'll look at qos-test during soft
> > freeze.
> >
> > 2) accept that I'll send another x86 pull request (not a large one)
> > after soft freeze, so that I have more time to debug this (likely
> > unrelated) build-oss-fuzz issue.
> 
> Either of these is fine; my requirement is only that either:
>  (1) the oss-fuzz gitlab CI job needs to in practice actually
> pass at least most of the time
>  (2) we need to switch it to ok-to-fail or disable it
> 
> so I don't have CI failing for every merge I make.

This is far from the first time that oss-fuzz has caused us pain. It
feels like it has been flaky  for prolonged periods of time, for as
long as it has existed.

When I tried to switch CI to use Fedora 35 oss-fuzz was consistently
failing for months for no obvious reason that I could determine
despite days of debugging. Then one day I woke up and it magically
started working again, for no obvious reason. Inexplicable.

Conceptually we benefit from fuzzing to find obscure bugs.
Have we actually found any real bugs from the oss-fuzz CI
job we have though ? Between us all, we've certainly lost
many many many days of developer time debugging the thing
for false failures.

The magic set of args needed to run it make it even more
unpleasant to deal with, and that scripts/oss-fuzz/build.sh
runs different code inside GitLab vs outside GitLab also
make debugging worse.

Personally I favour turning it into a non-gating job as
I don't want to invest more of my own time debugging
non-bugs in it.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|