On 12/29/2011 10:33 PM, Anthony Liguori wrote:
So I decided to do some snooping. Here are some stats:
address@hidden:~/git/autotest/client/tests/kvm/tests$ wc -l *.py
150 balloon_check.py
68 boot_savevm.py
190 cdrom.py
1875 cgroup.py
111 cpu_hotplug.py
170 enospc.py
71 floppy.py
72 getfd.py
89 hdparm.py
0 __init__.py
615 ksm_overcommit.py
107 migration_multi_host.py
117 migration.py
85 migration_with_file_transfer.py
43 migration_with_reboot.py
138 multi_disk.py
62 nic_bonding.py
104 nic_hotplug.py
60 nmi_watchdog.py
203 pci_hotplug.py
182 physical_resources_check.py
439 qemu_img.py
48 qemu_iotests.py
407 qmp_basic.py
389 qmp_basic_rhel6.py
54 set_link.py
67 smbios_table.py
356 stepmaker.py
247 steps.py
43 stop_continue.py
31 system_reset_bootable.py
181 timedrift.py
96 timedrift_with_migration.py
91 timedrift_with_reboot.py
103 timedrift_with_stop.py
28 unittest_kvmctl.py
121 unittest.py
90 usb.py
2174 virtio_console.py
85 vmstop.py
9562 total
There are more tests under client/virt/tests:
address@hidden tests]$ wc -l *.py
25 autotest.py
100 boot.py
42 build.py
32 clock_getres.py
235 ethtool.py
84 file_transfer.py
53 fillup_disk.py
76 guest_s4.py
80 guest_test.py
45 image_copy.py
0 __init__.py
136 iofuzz.py
31 ioquit.py
40 iozone_windows.py
127 jumbo.py
85 kdump.py
41 linux_s3.py
84 lvm.py
60 mac_change.py
46 module_probe.py
90 multicast.py
106 netperf.py
149 netstress_kill_guest.py
255 nfs_corrupt.py
63 nicdriver_unload.py
39 nic_promisc.py
73 ping.py
29 pxe.py
42 shutdown.py
147 softlockup.py
53 stress_boot.py
164 trans_hugepage_defrag.py
127 trans_hugepage.py
113 trans_hugepage_swapping.py
852 unattended_install.py
175 vlan.py
46 watchdog.py
136 whql_client_install.py
275 whql_submission.py
49 yum_update.py
4405 total
Not to mention the infrastructure libraries:
address@hidden virt]$ wc -l *.py
1351 aexpect.py
705 base_installer.py
9 common.py
0 __init__.py
195 installer.py
64 installer_unittest.py
240 kvm_installer.py
921 kvm_monitor.py
1635 kvm_vm.py
324 libvirt_monitor.py
1271 libvirt_vm.py
110 passfd.py
237 ppm_utils.py
519 rss_client.py
527 virt_env_process.py
124 virt_http_server.py
51 virt_passfd_setup.py
229 virt_scheduler.py
1401 virt_step_editor.py
131 virt_test.py
315 virt_test_setup.py
813 virt_test_utils.py
3556 virt_utils.py
196 virt_utils_unittest.py
224 virt_video_maker.py
1032 virt_vm.py
16180 total
There's a bit more outside this dir, not to mention the auxiliary data
files. The support libraries account for roughly for the same number
of lines of test code written, so we can say in very rough terms we
have around 33 k of python code for kvm/libvirt tests. I agree it is
more than an order of magnitude smaller than the qemu code base, but
still it demands effort that has to be justified.
address@hidden:~/git/autotest/client/tests$ git log --format="%an <%ae>"
kvm | sort -u
Amos Kong <address@hidden>
Chris Evich <address@hidden>
Cleber Rosa <address@hidden>
Jerry Tang <address@hidden>
lmr <address@hidden>
Lucas Meneghel Rodrigues <address@hidden>
Lucas Meneghel Rodrigues <address@hidden>
Lukas Doktor <address@hidden>
mbligh <address@hidden>
Onkar N Mahajan <address@hidden>
pradeep <address@hidden>
Qingtang Zhou <address@hidden>
Thomas Jarosch <address@hidden>
Yiqiao Pu <address@hidden>
Which leads me to the following conclusions:
1) No one outside of autotest developers is actually contributing tests
to kvm-autotest.
Autotest was maintained using subversion until recently, all commits
from people outside autotest were commited by autotest developers, the
git tree was just a mirror until recently. Grepping for Signed-off-by:
lines will give a much better idea of the contributors:
address@hidden autotest.git]$ git log client/virt | grep
"Signed-off-by:" | sort -u
Signed-off-by: Xu He Jie <address@hidden>
Signed-off-by: Alon Levy <address@hidden>
Signed-off-by: Amos Kong <address@hidden>
Signed-off-by: Chen Cao <address@hidden>
Signed-off-by: Chris Evich <address@hidden>
Signed-off-by: Cleber Rosa <address@hidden>
Signed-off-by: Feng Yang <address@hidden>
Signed-off-by: Gerd Hoffmann <address@hidden>
Signed-off-by: Jason D. Gaston <address@hidden>
Signed-off-by: Jason Wang <address@hidden>
Signed-off-by: Jiri Zupka <address@hidden>
Signed-off-by: Lucas Meneghel Rodrigues <address@hidden>
Signed-off-by: Lukas Doktor <address@hidden>
Signed-off-by: Martin Jenner <address@hidden>
Signed-off-by: Pradeep K Surisetty <address@hidden>
Signed-off-by: Pradeep Kumar Surisetty <address@hidden>
Signed-off-by: Qingtang Zhou <address@hidden>
Signed-off-by: Quentin Deldycke <address@hidden>
Signed-off-by: Ray Chauvin <address@hidden>
Signed-off-by: Satheesh Rajendran <address@hidden>
Signed-off-by: Suqin Huang <address@hidden>
Signed-off-by: Wei Yang <address@hidden>
Signed-off-by: Xu He Jie <address@hidden>
Signed-off-by: Yiqiao Pu <address@hidden>
Signed-off-by: Yogananth Subramanian <address@hidden>
Signed-off-by: Yolkfull Chow <address@hidden>
address@hidden autotest.git]$ git log client/tests/kvm | grep
"Signed-off-by:" | sort -u
Signed-off-by: Alon Levy <address@hidden>
Signed-off-by: Amos Kong <address@hidden>
Signed-off-by: Avi Kivity <address@hidden>
Signed-off-by: Bear Yang <address@hidden>
Signed-off-by: Cao, Chen <address@hidden>
Signed-off-by: Chen Cao <address@hidden>
Signed-off-by: Chris Evich <address@hidden>
Signed-off-by: Cleber Rosa <address@hidden>
Signed-off-by: David Huff <address@hidden>
Signed-off-by: Eduardo Habkost <address@hidden>
Signed-off-by: Feng Yang <address@hidden>
Signed-off-by: Gerd Hoffmann <address@hidden>
Signed-off-by: Jason Wang <address@hidden>
Signed-off-by: Jerry Tang <address@hidden>
Signed-off-by: Jes Sorensen <address@hidden>
Signed-off-by: Jin Bing Guo <address@hidden>
Signed-off-by: Jiri Zupka <address@hidden>
Signed-off-by: Ken Cao <address@hidden>
Signed-off-by: Lucas Meneghel Rodrigues <address@hidden
Signed-off-by: Luiz Capitulino <address@hidden>
Signed-off-by: Lukas Doktor <address@hidden>
Signed-off-by: Marcelo Tosatti <address@hidden>
Signed-off-by: Martin Jenner <address@hidden>
Signed-off-by: Michael Goldish <address@hidden>
Signed-off-by: Michael S. Tsirkin <address@hidden>
Signed-off-by: Onkar N Mahajan <address@hidden>
Signed-off-by: Pradeep K Surisetty <address@hidden>
Signed-off-by: Qingtang Zhou <address@hidden>
Signed-off-by: Ryan Harper <address@hidden>
Signed-off-by: Shuxi Shang <address@hidden>
Signed-off-by: Sudhir Kumar <address@hidden>
Signed-off-by: Supriya Kannery <address@hidden>
Signed-off-by: Suqin Huang <address@hidden>
Signed-off-by: Tang Chen <address@hidden>
Signed-off-by: Thomas Jarosch <address@hidden>
Signed-off-by: Uri Lublin <address@hidden>
Signed-off-by: Yiqiao Pu <address@hidden>
Signed-off-by: Yogananth Subramanian <address@hidden>
Signed-off-by: Yolkfull Chow <address@hidden>
So, not quite what you wanted to demonstrate.
2) Most of the tests are relatively small and simple enough that they
could be trivially ported to a stand alone utility. For instance,
looking at the pci_hotplug.py, it just executes monitor commands and
does basic pci enumeration in the guest.
They do depend on infrastructure code to be simple and small. The
infrastructure code was written using autotest libraries, so although
it is possible to make it stand alone, still there's the problem of
having these library/classes out of sync with autotest evolving.
I don't see a lot of pitfalls here. At the end of the day, we're not
talking about all that much code.
Even so (you see it's not so small), it demands effort that has to be
well justified, as I pointed out. I'm starting to believe it might be
worth the effort.
It's not about where they live, it's about the ability to execute them
in a simple and direct fashion.
Ok, fair enough, we'll see what we can do, although I wish it were a
simpler problem, see below.
You have a point with regards to having the test cases more stand
alone, but
it's not as easy as one would think mainly because of the large amount
of user
configurable options that we have to pass to the tests.
Configuration options are really the devil when it comes to testing. As
a developer, I want my test suite to just run and tell me if I have
bugs. I have no interest in configuring it to run a certain way. I don't
want to think that much about what I'm testing, I just want the tests to
run and tell me if I screwed something up.
Still, you need to tell your tests, say, the format of the terminal
prompt for Fedora might be slightly different of the
[insert-distro-here], that the src and destination paths on a file
transfer test are different for Windows and Linux guests, that the
output of a given command when you are running a test with a virtio
card differs from the output when you are using rtl8139, that you want
to build qemu from git repo X, or from brew build Y, using a custom
version of seabios, and so on and so forth.