qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU


From: Anthony Liguori
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Tue, 03 Jan 2012 07:52:43 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15

On 12/29/2011 07:20 PM, Lucas Meneghel Rodrigues wrote:
On 12/29/2011 10:33 PM, Anthony Liguori wrote:
So I decided to do some snooping. Here are some stats:

address@hidden:~/git/autotest/client/tests/kvm/tests$ wc -l *.py
150 balloon_check.py
68 boot_savevm.py
190 cdrom.py
1875 cgroup.py
111 cpu_hotplug.py
170 enospc.py
71 floppy.py
72 getfd.py
89 hdparm.py
0 __init__.py
615 ksm_overcommit.py
107 migration_multi_host.py
117 migration.py
85 migration_with_file_transfer.py
43 migration_with_reboot.py
138 multi_disk.py
62 nic_bonding.py
104 nic_hotplug.py
60 nmi_watchdog.py
203 pci_hotplug.py
182 physical_resources_check.py
439 qemu_img.py
48 qemu_iotests.py
407 qmp_basic.py
389 qmp_basic_rhel6.py
54 set_link.py
67 smbios_table.py
356 stepmaker.py
247 steps.py
43 stop_continue.py
31 system_reset_bootable.py
181 timedrift.py
96 timedrift_with_migration.py
91 timedrift_with_reboot.py
103 timedrift_with_stop.py
28 unittest_kvmctl.py
121 unittest.py
90 usb.py
2174 virtio_console.py
85 vmstop.py
9562 total

There are more tests under client/virt/tests:

address@hidden tests]$ wc -l *.py
25 autotest.py
100 boot.py
42 build.py
32 clock_getres.py
235 ethtool.py
84 file_transfer.py
53 fillup_disk.py
76 guest_s4.py
80 guest_test.py
45 image_copy.py
0 __init__.py
136 iofuzz.py
31 ioquit.py
40 iozone_windows.py
127 jumbo.py
85 kdump.py
41 linux_s3.py
84 lvm.py
60 mac_change.py
46 module_probe.py
90 multicast.py
106 netperf.py
149 netstress_kill_guest.py
255 nfs_corrupt.py
63 nicdriver_unload.py
39 nic_promisc.py
73 ping.py
29 pxe.py
42 shutdown.py
147 softlockup.py
53 stress_boot.py
164 trans_hugepage_defrag.py
127 trans_hugepage.py
113 trans_hugepage_swapping.py
852 unattended_install.py
175 vlan.py
46 watchdog.py
136 whql_client_install.py
275 whql_submission.py
49 yum_update.py
4405 total

Hrm, what's the significance of the two separate directories?

At any rate, my point still stands. 13k LOC is still small enough that making large changes to the infrastructure shouldn't be that big of a deal. I think the important observation is that each test is pretty small which means they can be reasonably moved 1-by-1 to a new stand alone infrastructure.

Not to mention the infrastructure libraries:

address@hidden virt]$ wc -l *.py
1351 aexpect.py
705 base_installer.py
9 common.py
0 __init__.py
195 installer.py
64 installer_unittest.py
240 kvm_installer.py
921 kvm_monitor.py
1635 kvm_vm.py
324 libvirt_monitor.py
1271 libvirt_vm.py
110 passfd.py
237 ppm_utils.py
519 rss_client.py
527 virt_env_process.py
124 virt_http_server.py
51 virt_passfd_setup.py
229 virt_scheduler.py
1401 virt_step_editor.py
131 virt_test.py
315 virt_test_setup.py
813 virt_test_utils.py
3556 virt_utils.py
196 virt_utils_unittest.py
224 virt_video_maker.py
1032 virt_vm.py
16180 total

There's a bit more outside this dir, not to mention the auxiliary data files.
The support libraries account for roughly for the same number of lines of test
code written, so we can say in very rough terms we have around 33 k of python
code for kvm/libvirt tests. I agree it is more than an order of magnitude
smaller than the qemu code base, but still it demands effort that has to be
justified.

address@hidden:~/git/autotest/client/tests$ git log --format="%an <%ae>"
kvm | sort -u

Amos Kong <address@hidden>
Chris Evich <address@hidden>
Cleber Rosa <address@hidden>
Jerry Tang <address@hidden>
lmr <address@hidden>
Lucas Meneghel Rodrigues <address@hidden>
Lucas Meneghel Rodrigues <address@hidden>
Lukas Doktor <address@hidden>
mbligh <address@hidden>
Onkar N Mahajan <address@hidden>
pradeep <address@hidden>
Qingtang Zhou <address@hidden>
Thomas Jarosch <address@hidden>
Yiqiao Pu <address@hidden>

Which leads me to the following conclusions:

1) No one outside of autotest developers is actually contributing tests
to kvm-autotest.

Autotest was maintained using subversion until recently, all commits from people
outside autotest were commited by autotest developers, the git tree was just a
mirror until recently. Grepping for Signed-off-by: lines will give a much better
idea of the contributors:

address@hidden autotest.git]$ git log client/virt | grep "Signed-off-by:" | 
sort -u
Signed-off-by: Xu He Jie <address@hidden>
Signed-off-by: Alon Levy <address@hidden>
Signed-off-by: Amos Kong <address@hidden>
Signed-off-by: Chen Cao <address@hidden>
Signed-off-by: Chris Evich <address@hidden>
Signed-off-by: Cleber Rosa <address@hidden>
Signed-off-by: Feng Yang <address@hidden>
Signed-off-by: Gerd Hoffmann <address@hidden>
Signed-off-by: Jason D. Gaston <address@hidden>
Signed-off-by: Jason Wang <address@hidden>
Signed-off-by: Jiri Zupka <address@hidden>
Signed-off-by: Lucas Meneghel Rodrigues <address@hidden>
Signed-off-by: Lukas Doktor <address@hidden>
Signed-off-by: Martin Jenner <address@hidden>
Signed-off-by: Pradeep K Surisetty <address@hidden>
Signed-off-by: Pradeep Kumar Surisetty <address@hidden>
Signed-off-by: Qingtang Zhou <address@hidden>
Signed-off-by: Quentin Deldycke <address@hidden>
Signed-off-by: Ray Chauvin <address@hidden>
Signed-off-by: Satheesh Rajendran <address@hidden>
Signed-off-by: Suqin Huang <address@hidden>
Signed-off-by: Wei Yang <address@hidden>
Signed-off-by: Xu He Jie <address@hidden>
Signed-off-by: Yiqiao Pu <address@hidden>
Signed-off-by: Yogananth Subramanian <address@hidden>
Signed-off-by: Yolkfull Chow <address@hidden>

address@hidden autotest.git]$ git log client/tests/kvm | grep "Signed-off-by:" |
sort -u
Signed-off-by: Alon Levy <address@hidden>
Signed-off-by: Amos Kong <address@hidden>
Signed-off-by: Avi Kivity <address@hidden>
Signed-off-by: Bear Yang <address@hidden>
Signed-off-by: Cao, Chen <address@hidden>
Signed-off-by: Chen Cao <address@hidden>
Signed-off-by: Chris Evich <address@hidden>
Signed-off-by: Cleber Rosa <address@hidden>
Signed-off-by: David Huff <address@hidden>
Signed-off-by: Eduardo Habkost <address@hidden>
Signed-off-by: Feng Yang <address@hidden>
Signed-off-by: Gerd Hoffmann <address@hidden>
Signed-off-by: Jason Wang <address@hidden>
Signed-off-by: Jerry Tang <address@hidden>
Signed-off-by: Jes Sorensen <address@hidden>
Signed-off-by: Jin Bing Guo <address@hidden>
Signed-off-by: Jiri Zupka <address@hidden>
Signed-off-by: Ken Cao <address@hidden>
Signed-off-by: Lucas Meneghel Rodrigues <address@hidden
Signed-off-by: Luiz Capitulino <address@hidden>
Signed-off-by: Lukas Doktor <address@hidden>
Signed-off-by: Marcelo Tosatti <address@hidden>
Signed-off-by: Martin Jenner <address@hidden>
Signed-off-by: Michael Goldish <address@hidden>
Signed-off-by: Michael S. Tsirkin <address@hidden>
Signed-off-by: Onkar N Mahajan <address@hidden>
Signed-off-by: Pradeep K Surisetty <address@hidden>
Signed-off-by: Qingtang Zhou <address@hidden>
Signed-off-by: Ryan Harper <address@hidden>
Signed-off-by: Shuxi Shang <address@hidden>
Signed-off-by: Sudhir Kumar <address@hidden>
Signed-off-by: Supriya Kannery <address@hidden>
Signed-off-by: Suqin Huang <address@hidden>
Signed-off-by: Tang Chen <address@hidden>
Signed-off-by: Thomas Jarosch <address@hidden>
Signed-off-by: Uri Lublin <address@hidden>
Signed-off-by: Yiqiao Pu <address@hidden>
Signed-off-by: Yogananth Subramanian <address@hidden>
Signed-off-by: Yolkfull Chow <address@hidden>

So, not quite what you wanted to demonstrate.

Well it actually still is. There isn't a tremendous overlap between that list and the list of QEMU contributors.

2) Most of the tests are relatively small and simple enough that they
could be trivially ported to a stand alone utility. For instance,
looking at the pci_hotplug.py, it just executes monitor commands and
does basic pci enumeration in the guest.

They do depend on infrastructure code to be simple and small. The infrastructure
code was written using autotest libraries, so although it is possible to make it
stand alone, still there's the problem of having these library/classes out of
sync with autotest evolving.

I don't see a lot of pitfalls here. At the end of the day, we're not
talking about all that much code.

Even so (you see it's not so small), it demands effort that has to be well
justified, as I pointed out. I'm starting to believe it might be worth the 
effort.

There are two discussions tied into one here so just to summarize:

1) I think we can get more people working on kvm autotest and by extracting and making them stand alone. We can potentially even start merging tests into QEMU getting more developers working on kvm autotest tests and mandating new ones for certain types of features.

2) kvm autotest focuses on producing guest-OS agnostic tests which require infrastructure (Python, ssh, etc.). qemu-test essentially uses Linux as a libOS for writing unit tests. (1) does not change the need for qemu-test IMHO.

It's not about where they live, it's about the ability to execute them
in a simple and direct fashion.

Ok, fair enough, we'll see what we can do, although I wish it were a simpler
problem, see below.

You have a point with regards to having the test cases more stand
alone, but
it's not as easy as one would think mainly because of the large amount
of user
configurable options that we have to pass to the tests.

Configuration options are really the devil when it comes to testing. As
a developer, I want my test suite to just run and tell me if I have
bugs. I have no interest in configuring it to run a certain way. I don't
want to think that much about what I'm testing, I just want the tests to
run and tell me if I screwed something up.

Still, you need to tell your tests, say, the format of the terminal prompt for
Fedora might be slightly different of the [insert-distro-here], that the src and
destination paths on a file transfer test are different for Windows and Linux
guests,

There are various clever things we can do. Essentially, the problem is that we want to deliver a test payload to a guest, right? Why not create a simple PCI device and expose the payload through the bar or something? There's no need to screen scrap a terminal session when we can do something more direct.

that the output of a given command when you are running a test with a
virtio card differs from the output when you are using rtl8139, that you want to
build qemu from git repo X, or from brew build Y, using a custom version of
seabios, and so on and so forth.

Building qemu should not be tied into the tests themselves. That's useful for regression testing but that just complicates things for development testing.

The building part is somethign that probably should continue to live in 
autotest.

Regards,

Anthony Liguori






reply via email to

[Prev in Thread] Current Thread [Next in Thread]