Re: [Qemu-devel] [PATCH 2/2] test: replace gtester with a TAP driver

[Daniel P . Berrangé]

On Thu, Nov 29, 2018 at 06:45:31PM +0100, Paolo Bonzini wrote:
> gtester is deprecated by upstream glib and it does not support tests
> that call g_test_skip in some glib stable releases.
> 
> glib suggests instead using Automake's TAP support.  We do not support
> Automake, but we can copy the code that beautifies the TAP output and
> use it.  I chose to use the Perl copy rather than the shell/awk one,
> in order to reuse Perl's TAP parsing package, but I'm open to suggestions
> about which language to use.

Do we even need the beutifier ?

IIUC, this output is only seen when you with 'make check' passing V=1,
so most people won't see it.

When V=1 is set, it just changes:

$ ./tests/test-io-channel-socket -k --tap
# random seed: R02Sb9d02d2a9658ae1be211540f5ceebc01
1..9
# Start of io tests
# Start of channel tests
# Start of socket tests
ok 1 /io/channel/socket/ipv4-sync
ok 2 /io/channel/socket/ipv4-async
ok 3 /io/channel/socket/ipv4-fd
ok 4 /io/channel/socket/ipv6-sync
ok 5 /io/channel/socket/ipv6-async
ok 6 /io/channel/socket/unix-sync
ok 7 /io/channel/socket/unix-async
**
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)
Bail out! 
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)
Aborted (core dumped)


To look like:

$ ./tests/test-io-channel-socket -k --tap | ./scripts/tap-driver.pl 
PASS 1 /io/channel/socket/ipv4-sync
PASS 2 /io/channel/socket/ipv4-async
PASS 3 /io/channel/socket/ipv4-fd
PASS 4 /io/channel/socket/ipv6-sync
PASS 5 /io/channel/socket/ipv6-async
PASS 6 /io/channel/socket/unix-sync
PASS 7 /io/channel/socket/unix-async
ERROR - Bail out! 
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)

where 'PASS' is coloured in green and 'ERROR' is red. I'm not
convinced this change justifies having the tap-driver.pl script

Since this commit already proposes a dep on the perl-Test-Harness
package, the host will have the 'prove' command, which can be used
for invoking tests.

Prove just needs a wrapper to know how to invoke glib tests:

  $ cat glibharness.sh 
  #!/bin/sh

  exec "$@" -k --tap

Now you can run :

$ prove -v -e ./glibharness.sh  tests/test-io-channel-socket
tests/test-io-channel-socket .. 
# random seed: R02S7b47ca0de1bc27cbbcf22dcc130c2b7f
1..9
# Start of io tests
# Start of channel tests
# Start of socket tests
ok 1 /io/channel/socket/ipv4-sync
ok 2 /io/channel/socket/ipv4-async
ok 3 /io/channel/socket/ipv4-fd
ok 4 /io/channel/socket/ipv6-sync
ok 5 /io/channel/socket/ipv6-async
ok 6 /io/channel/socket/unix-sync
ok 7 /io/channel/socket/unix-async
**
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)
Bailout called.  Further testing stopped:  
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)
Bail out! 
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)
FAILED--Further testing stopped: 
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)


Where the failure line is now coloured in Red (or sucess is coloured
in green)

prove can also run tests in batches, giving progress of execution of
each test as it runs

$ prove  -e ./glibharness.sh  
tests/test-io-channel-{buffer,command,file,socket,tls} 
tests/test-io-channel-buffer ... ok   
tests/test-io-channel-command .. ok   
tests/test-io-channel-file ..... ok   
tests/test-io-channel-socket ... 1/? **
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)
Bailout called.  Further testing stopped:  
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)
FAILED--Further testing stopped: 
ERROR:tests/test-io-channel-socket.c:424:test_io_channel_unix_fd_pass: 
assertion failed: (0)


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|