Re: Ramping up Continuous Fuzzing of Virtual Devices in QEMU

[Alexander Bulekov]

On 201104 1546, Peter Maydell wrote:
> On Wed, 4 Nov 2020 at 15:26, Alexander Bulekov <alxndr@bu.edu> wrote:
> > If I understand correctly, this is analogous to what happens with
> > Coverity reports. Access to Coverity is closed (not sure if there is a
> > process to apply for access). It also seems that there is a push to fix
> > CID issues prior to new releases. Maybe a similar process can be used for
> > fuzzing?
> 
> Coverity is only closed in the sense that you have to request
> an account on the website. Anybody who's a QEMU developer
> can look at the reports.
> 

OK thats good to know. Why should fuzzing reports be treated
differently? Is it because they come with a ready-to-use reproducer?

> The attempt to fix CID issues works because:
>  * Coverity reports a fairly small number of issues, so
>    "fix them all" is relatively tractable, and then once you
>    get down to "no outstanding issues" the only new ones
>    that get found are for new changes to QEMU (not entirely
>    true, but close enough)

I think fuzzing is quite similar. After an initial wave of reports,
they should slow to a trickle and eventually should just catch problems
in new changes.

>  * Mostly issues are reported soon after the offending code
>    goes into the tree, so it's often possible to quickly
>    identify the patch that introduced the issue and ask
>    the person who wrote that patch to fix the bug

While fuzzing can take some time to find issues after new code is
committed, I don't think the problem of reports on stale code is unique
to fuzzing. That should only be an issue for the initial wave of reports
(the same as I'm guessing it was was when Coverity started analyzing
QEMU in 2013).
Additionally, we can filter crashes based on where they occured, which
should give a similar level of control as the Coverity
components/patterns. If problems in e.g. ati or vmxnet are unlikely to
be looked at, we can
 1.) Not fuzz them (easy to do by removing an entry from
 tests/qtest/fuzz/generic_fuzz_configs.h)
 2.) Apply filters to ignore crash reports with ati.c or vmxnet.c in the
 callstack.

That said, IIUC main purpose of the Coverity componsnets/patterns is to
prevent false positives. This shouldn't be nearly as much of a problem
for oss-fuzz reports for which we can provide qtest reproducers.

>  * Coverity reports are categorized by kind-of-failure,
>    so it's easy to prioritize important stuff (buffer overflows)
>    and leave less significant stuff (dead code) for later

Fuzzer reports are the same. OSS-fuzz tells us if something is an
overflow, UAF, overlapping-memcpy, double-free, assertion-failure,
null-ptr derefs etc.
If noise from assertion-failure/null-ptr derefs reports is a concern,
maybe we can just ignore those for now?

>  * Coverity's reports include the automated analysis of
>    why Coverity thinks there's an issue -- this is not
>    always right but it's a solid head start on "what's the
>    bug here" compared to just having a repro case and an
>    assertion-failure message

Thats true - the types of bugs found by fuzzing rather than static
analysis usually are tough to automatically suggest (pretty) fixes for.
I have been thinking about ways to make this situation better, beyond
just enabling the relevant -trace events and hoping they provide context
and make life easier.

>  * There's a set of people who care enough about Coverity
>    reports to put the time in to fixing them...

I hope this set of people grows for fuzzing reports as well. Though
fuzzing reports are nothing compared to the ~1250 fixed coverity
defects, fuzzing has already helped highlight some serious issues that
have been hiding in the code for a long time. Unfortunately, I think
there is a gap between the types of problems reported by fuzzing and
Coverity, where I don't see someone picking up a dozen random fuzzing
reports and having a dozen patches ready by the end of the day. Multiple
people working on fuzzer-discovered issues have mentioned that they are
often quite time consuming to properly fix (for little preceived
reward). Maybe the solution is to limit the scope of fuzzer reports so
that they might be rarer, but are more likely to feature problems that
people will care about? This might mean having a narrower selection of
fuzzed devices than just "anything that works with KVM"

-Alex

> 
> thanks
> -- PMM