Re: [lmi] Better error messages for the GUI test failures

[Murphy, Kimberly]

Vadim Zeitlin wrote:

>  Here is the promised patch improving the error messages in case of GUI
> test failures. A large(r) part of these changes is actually in wxWidgets
> itself and so you need to use 0eca25d395c384fa6ecfaa7120e3be0c129c618a
> (r78416) or later for this to work and, actually, with that wxWidgets
> version it should already be possible to identify what exactly fails in
> the
> validate_output_illustration test on Kimberley's machine, but the patch
> makes it even easier by adding unique descriptive messages to almost all
> wxTEST_DIALOG() calls ("almost" because I didn't modify those that will
> need to be changed in the near future anyhow).
> [...] 
>  Kimberley, could you please let me know what is the new output for the
> output validation tests on your machine with these changes, hopefully
> this
> will allow us to diagnose them.

Using revision 6109 along with the updated wx trunk designated 
here:
 http://lists.nongnu.org/archive/html/lmi/2015-01/msg00043.html

the validate output tests no longer fail:

/opt/lmi/bin[0]$./wx_test.exe --mellon \
--gui_test_path=/opt/lmi/gui_test --data_path=/opt/lmi/data 
NOTE: starting the test suite
about_dialog_version: started
About dialog version string is "20150116T1757Z".
time=26ms (for about_dialog_version)
about_dialog_version: ok
benchmark_census: started
Run case for MSEC_EGL_20040729.cns: 177ms elapsed
Print case to PDF for MSEC_EGL_20040729.cns: 13956ms elapsed
Print case to spreadsheet for MSEC_EGL_20040729.cns: 809ms elapsed
Run case for MSEC_FIELD_20040729.cns: 249ms elapsed
Print case to PDF for MSEC_FIELD_20040729.cns: 11791ms elapsed
Print case to spreadsheet for MSEC_FIELD_20040729.cns: 623ms elapsed
Run case for MSEC_SL10_20040729.cns: 184ms elapsed
Print case to PDF for MSEC_SL10_20040729.cns: 14789ms elapsed
Print case to spreadsheet for MSEC_SL10_20040729.cns: 939ms elapsed
Run case for MSEC_SL14s_20040729.cns: 206ms elapsed
Print case to PDF for MSEC_SL14s_20040729.cns: 15766ms elapsed
Print case to spreadsheet for MSEC_SL14s_20040729.cns: 843ms elapsed
time=61007ms (for benchmark_census)
benchmark_census: ok
...
validate_output_illustration: started
time=1639ms (for validate_output_illustration)
validate_output_illustration: ok
validate_output_mec: started
time=204ms (for validate_output_mec)
validate_output_mec: ok
time=82169ms (for all tests)
SUCCESS: 20 tests successfully completed.
NOTE: 3 tests were skipped

This is welcomed news. Though, I question what might have been the 
problem and what may have fixed it. In order to view the new 
descriptive messages, I need the test to fail. 

Backtracking, the validate output tests had failed for me with 
revision 6086 (and the previously announced wx trunk). So, using 
the updated wx along with revision 6086, I reran the automated 
suite. The validate output tests do indeed fail:

/opt/lmi/bin[0]$./wx_test.exe --mellon \
--gui_test_path=/opt/lmi/gui_test --data_path=/opt/lmi/data 
NOTE: starting the test suite
about_dialog_version: started
About dialog version string is "20150116T1757Z".
time=32ms (for about_dialog_version)
about_dialog_version: ok
benchmark_census: started
Run case for MSEC_EGL_20040729.cns: 179ms elapsed
Print case to PDF for MSEC_EGL_20040729.cns: 14099ms elapsed
Print case to spreadsheet for MSEC_EGL_20040729.cns: 815ms elapsed
Run case for MSEC_FIELD_20040729.cns: 246ms elapsed
Print case to PDF for MSEC_FIELD_20040729.cns: 12093ms elapsed
Print case to spreadsheet for MSEC_FIELD_20040729.cns: 622ms elapsed
Run case for MSEC_SL10_20040729.cns: 183ms elapsed
Print case to PDF for MSEC_SL10_20040729.cns: 14869ms elapsed
Print case to spreadsheet for MSEC_SL10_20040729.cns: 947ms elapsed
Run case for MSEC_SL14s_20040729.cns: 206ms elapsed
Print case to PDF for MSEC_SL14s_20040729.cns: 15382ms elapsed
Print case to spreadsheet for MSEC_SL14s_20040729.cns: 815ms elapsed
time=61133ms (for benchmark_census)
benchmark_census: ok
...
validate_output_illustration: started
validate_output_illustration: ERROR (Assertion failure: Expected 
wxMessageDialog with wxYES_NO style was not shown. [file 
/opt/lmi/src/lmi/wx_test_new.hpp, line 84, in close_discard_changes()].)
validate_output_mec: started
validate_output_mec: ERROR (Assertion failure: Expected dialog of type 
"13MvcController" was not shown. [file 
/opt/lmi/src/lmi/wx_test_validate_output.cpp, line 200, in run()].)
time=83591ms (for all tests)
FAILURE: 2 out of 23 tests failed.
NOTE: 3 tests were skipped

The messages have improved. A number of improvements were made 
between revision 6086 and 6109. Can we pinpoint what the issue 
might have been? 



This e-mail transmission may contain information that is proprietary, 
privileged and/or confidential and is intended exclusively for the person(s) to 
whom it is addressed. Any use, copying, retention or disclosure by any person 
other than the intended recipient or the intended recipient's designees is 
strictly prohibited. If you are not the intended recipient or their designee, 
please notify the sender immediately by return e-mail and delete all copies.