Re: [lmi] Automated GUI testing, revisited

[Vadim Zeitlin]

On Wed, 12 Nov 2014 12:18:19 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2014-11-12 01:55Z, Greg Chicares wrote:
GC> [...]
GC> > Vadim--I believe I copied 'wx_test.conf' from an old email that
GC> > you had sent. Was "time_disk=11000" an actual measurement? For
GC> > single tasks, your machine should be faster than mine; so I'm
GC> > wondering whether we're running the same test.
GC> > 
GC> > My "456ms" above looks a lot like "time_run" in 'wx_test.conf':
GC> >   time_run=434
GC> >   time_disk=11000
GC> >   time_spreadsheet=710
GC> > (I mention that without further investigation on my part).
GC> 
GC> Here's more information, from another run:
GC> 
GC> 11:49:25: Run case for census_1: 416ms elapsed (expected -4.15%)
GC> 11:49:25:     benchmark_census: ERROR (Assertion 
'std::fabs(diff_in_percents) < 10' failed
GC> (expected 11000ms, but actually took 416ms, i.e. -96.22%).
GC> [file /lmi/src/lmi/wx_test_benchmark_census.cpp, line 101] )
GC> 
GC> Doesn't this suggest that the "ERROR" comparison is incorrect?

 Sorry if I'm missing something, but I don't see anything wrong with it.

GC> I can match "expected -4.15%" this way:
GC>   (416-434)/434 = -0.04147...
GC> so that seems right.

 Yes.

GC> Assuming that "416ms" is my "time_run",

 Just to be clear: this is true assuming you mean "the actual time taken by
running the test" by "your time_run". This is not the same as "time_run
from your config file" which is "the expected time for running the test".

 Sorry if this seems completely tautological, but I'm just trying to
understand where could the misunderstanding be coming from and so would
like to make all the terms as precise as possible.

GC> I can match "i.e. -96.22%" this way:
GC>   (416-11000)/11000 = -0.96218...
GC> but it seems incorrect to compare observed "time_run" to
GC> expected "time_disk", if that's what's really being done.

 No, by some coincidence (?) printing the case to disk also takes 416ms on
your machine. I think this is wrong and that the test actually did detect a
problem as IME the time for printing the test is much longer than for just
running it. I'm not sure if you're running the version in which
Ctrl-Shift-K is used for printing to disk or one in which it was already
changed to Ctrl-Shift-I, the latter would explain it.

 In any case, -96.22% does come from the expression above, but 416 is not
"time_run" in it, it's, to quote the test output above, time it actually
took to run "Print to disk" command.


 Hope this clears it up,
VZ