Re: [PATCH 07/11] iotests/264: move to python unittest

[Vladimir Sementsov-Ogievskiy]

21.01.2021 04:17, Eric Blake wrote:
> On 11/18/20 12:04 PM, Vladimir Sementsov-Ogievskiy wrote:
> > We are going to add more test cases, so use the library supporting test
> > cases.
> >
> > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> > ---
> >   tests/qemu-iotests/264     | 93 ++++++++++++++++++++++----------------
> >   tests/qemu-iotests/264.out | 20 ++------
> >   2 files changed, 58 insertions(+), 55 deletions(-)
>
> > +++ b/tests/qemu-iotests/264.out
> > @@ -1,15 +1,5 @@
> > -Start NBD server
> > -{"execute": "blockdev-add", "arguments": {"driver": "raw", "file": {"driver": "nbd", "reconnect-delay": 10, "server": {"path": 
> > "TEST_DIR/PID-nbd-sock", "type": "unix"}}, "node-name": "backup0"}}
> > -{"return": {}}
> > -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "speed": 1048576, "sync": "full", 
> > "target": "backup0"}}
> > -{"return": {}}
> > -Backup job is started
> > -Kill NBD server
> > -Backup job is still in progress
> > -{"execute": "block-job-set-speed", "arguments": {"device": "drive0", "speed": 
> > 0}}
> > -{"return": {}}
> > -Start NBD server
> > -Backup completed: 5242880
> > -{"execute": "blockdev-del", "arguments": {"node-name": "backup0"}}
> > -{"return": {}}
> > -Kill NBD server
> > +.
> > +----------------------------------------------------------------------
> > +Ran 1 tests
> > +
> > +OK
>
> I find it a shame that the expected output no longer shows what was
> executed.  But the test still passes, and if it makes it easier for you
> to extend the test in a later patch, I won't stand in the way (this is
> more an indication that by stripping the useful output, I'm no longer in
> as decent a position to help debug if the test starts failing).

Still, what is executed is understandable from the test itself.. And IMHO, 
debugging python unittests is simpler: you get the stack, and immediately see 
what happens. When with output-checking tests, you should first understand, 
what is the statement corresponding to the wrong output. It's not saying about 
the fact that with unittests I can simply test only one test-case (that's the 
reason, why I think that tests with several testcases must be written as 
unittest tests). And debugging output-checking tests with a lot of test-cases 
inside is always a pain for me.

Another benefit of unittest: on failure test immediately finishes. With 
output-checking tests, test continue to execute, and may produce unnecessary 
not-matching log, or hang, or anything else.

Another drawback of output-cheking tests: they often test too much unrelated 
things. Sometimes it's good: you can catch some unrelated bug :) But often it's 
a pain: you have to modify test outputs when creating new features or changing 
the behaviour actually unrelated to what the test actually want to test.

Python unittests are more difficult to write, as you should understand what 
exactly you want/should to check.. When with output-checking tests you can just 
log everything. But in general I'm for python unittests.

Still I think sometimes about supporting output for python-unitests based tests 
(not loosing the ability to execute test-cases in separate, may be .out file 
per test-case?), it may be a good compromise.

> Reviewed-by: Eric Blake <eblake@redhat.com>


--
Best regards,
Vladimir