Re: [lmi] Automated GUI testing, revisited

[Greg Chicares]

On 2014-11-09 01:21Z, Greg Chicares wrote:
> Now that we have an automated GUI test, I'm starting this new thread
> to ask some questions about what it's intended to do, and to discuss
> where we should go from here.

I propose that 'wx_test' run the commands on the "Test" menu. It is
my understanding that they're run manually today, so they seem to be
natural candidates for automation. If they're all run automatically,
then I see no need for the "Test" menu to continue to exist.

"Test system command..." should be run with a variety of commands.

(a) A command that always succeeds.

(b) A command that fails, e.g.:
  Exit code 2 from command 'ls xx'.
  Errors:
  ls: cannot access xx: No such file or directory

(c) An unrecognized command:
  Command 'foo' not recognized.
(In this case,
  15:30:13: Error: Execution of command 'foo' failed (error 2: the system 
cannot find the file specified.)
appears on the console, but I don't see any need to check that.)

(d) An empty string--which, if it occurred in production, would
probably identify a mistake.

[letters e-o reserved for any ideas others wish to add]

"Test date conversions" takes about four minutes to run. We don't
want to run it often. But we must run it occasionally--when we
upgrade wx or change lmi's class calendar_date--because any error
it finds may indicate a grave problem. Here are two ideas:

(p) Have 'wx_test' invoke it, but make it depend on the proposed
'--distribution' option. Then it's sure to be run before any
release. And this seems easy to code.

(q) First, recognize its nature: it tests a non-GUI part of wx.
Therefore, it doesn't belong in a GUI test. It doesn't belong with
lmi's 'unit_tests' target either, because it requires wx. This sui
generis test deserves special treatment. To cover changes to lmi's
class calendar_date, it's enough to preserve the ability to run it
at will; perhaps it should become a command-line option if there's
no other reason to keep the "Test" menu. Covering wx upgrades is
trickier. We could run it as a final step in 'install_wx.make',
except for the fatal problem that lmi is not necessarily available
at that point. For some additional expenditure of cleverness, we
might set up a sentinel file to cause it to be run whenever we
rebuild wx. The more I think about this, the better (p) seems.