Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Aleksandar Markovic]

On Wed, Jan 22, 2020 at 12:28 PM Stefan Hajnoczi <address@hidden> wrote:
>
> On Tue, Jan 21, 2020 at 03:07:53PM +0100, Aleksandar Markovic wrote:
> > On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <address@hidden> wrote:
> > >
> > > On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
> > > > 3) The community will be given all devised performance measurement 
> > > > methods in the form of easily reproducible step-by-step setup and 
> > > > execution procedures.
> > >
> > > Tracking performance is a good idea and something that has not been done
> > > upstream yet.
> >
> > Thanks for the interest, Stefan!
> >
> > >  A few questions:
> > >
> > >  * Will benchmarks be run automatically (e.g. nightly or weekly) on
> > >    someone's hardware or does every TCG architecture maintainer need to
> > >    run them manually for themselves?
> >
> > If the community wants it, definitely yes. Once the methodology is
> > developed, it should be straightforward to setup nightly and/or weekly
> > benchmarks - that could definitely include sending mails with reports
> > to the entire list or just individuals or subgroups. The recipient
> > choice is just a matter or having decent criteria about
> > appropriateness of information within the message (e.g. not to flood
> > the list with the data most people are not really interested).
> >
> > For linux-user tests, they are typically very quick, and nightly tests
> > are quite feasible to run. On someone hardware, of course, and
> > consistently always on the same hardware, if possible. If it makes
> > sense, one could setup multiple test beds with a variety of hardware
> > setups.
> >
> > For system mode tests, I knoe they are much more difficult to
> > automate, and, on top of that, there could be greater risk of
> > hangs/crashes Also, considering the number of machines we support,
> > those tests could consume much more time - perhaps even one day would
> > not be sufficient, if we have many machines and boot/shutdown
> > variants. For these reason, perhaps weekly executions would be more
> > appropriate for them, and, in general, given greater complexity, the
> > expectation from system-mode performance tests should be better kept
> > quite low for now.
> >
> > >  * Where will the benchmark result history be stored?
> > >
> >
> > If emailing is set up, the results could be reconstructed from emails.
> > But, yes, it would be better if the result history is kept somewhere
> > on an internet-connected file server
>
> Thanks.  I don't want to overcomplicate this project.  The main thing is
> to identify the stakeholders (TCG target maintainers?) and make sure
> they are happy.
>

Yes, Stefan, TCG target maintainers would be the main stakeholders. To
some extent, various Machine maintainers would also be stakeholders,
but they will most likely come back to TCG target maintainers looking
for solution. In a literal sense, a number of maintainers were
initially going to be very unhappy seeing the results (for example,
seeing that the machine or entire target performs poorly compared to
similar machines/targets), but after a while they should and will
become happy realizing the problem was identified, and the culprit is
at least approximately determined.

I intentionally wanted to keep the project description simple in order
to be realistic and not develop high expectation among any of us. And
if the student proved to be capable, it will be very easy to add some
more useful tasks for him in this area, to be included in his/hers
GSoC/Outreachy activities.

He had just today one case of performance degradation identified manually:

https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg06326.html

This project aims to do these kind of things easier, and possibly in
an automated way. Howard did this by manual measurements for one
particular setup, but this project will cover much much more.

Thanks, Stefan, again for your interest - and everything else!

Aleksandar



> Stefan