Author: Jay P. VanSant (jayv@portland.parcplace.com)
At the Portland ParcPlace Systems office in 1993, we began a large-scale application development project on contract with a consulting company. The project used VisualWorks, ParcPlace's Smalltalk development environment. We researched available testing tools, seeking to find a tool that would enable cross-platform testing on Windows, Mac, and UNIX. We developed a set of requirements and evaluated several available products against those requirements. One of the key requirements was that test scripts or recordings work without modification between platforms. We identified only two tools that use an object-oriented approach to scripting or recording the events in relation to the underlying GUI widgets (getting their locations and attributes at run time from the underlying API). Only one of the tools was able to interface to VisualWorks, and it turned out to not be sufficiently extensible to handle all of the custom widgets our application was using.
Consequently, we proceeded to develop and use a testing framework constructed within VisualWorks itself, using extensions to the Smalltalk class library as a test scripting language. The general testing framework, called TestWorks, works with a test class library called GUITester, and with a test coverage analyzer called TCA.
I will present our requirements for a testing environment, and then describe how TestWorks, TCA, and GUITester met these requirements. The design concepts used in building the tools will be presented, and then user interface and usage will be described. I will talk about our experience in building and using the tools, and ways that I would improve them based on that experience.
A short description of the tools:
TestWorks is a framework for creating and running test methods written in Smalltalk. It provides a full browser to locate classes and see them in relation to the inheritance hierarchy. Test methods are stored on the class side of a class in a protocol called TestWorks. Log files hold the list of test methods in a test suite, the results (logs) of executing the current test (pass-fail status, logged comments, stored objects for test comparison, and execution time), and a baseline log that can be used to compare current results with stored baseline results. The log files are stored in a specified directory and can be run sequentially as a regression test. A summary report covering all of the test suites included in the regression test is available. The directory of log files can also be stored as literal arrays on a TestWorksDirectory class, to allow configuration/version management of tests and test results together with the software under test. TCA is a test coverage analyzer that measures branch coverage. The user includes selected classes and methods in a list, and then TCA recompiles the object code (not the source) to have counters at each branch. Once this is done, tests can be run, and the report updated to show which methods are hit, how many times they are hit, and the percentage of branches in each method that have been exercised. The tester can view the actual methods to see which branches are not hit and then redesign the test suites to extend the coverage. The list of classes and methods can be stored with the TestWorks log file, so the same TCA setup can easily be created and used during regression tests.
GUITester includes a class library that corresponds with the VisualWorks SpecWrapper classes that specify widgets in the GUI, and a GUI event engine class that creates the basic user events. In each test class, methods allow a test writer to manipulate that widget appropriately for its type, such as clicking on action buttons, or making selections in lists. These methods actually create keyboard and mouse events directed to the widgets. In a typical test method, the test writer creates an instance of a GUITester, then initializes it with a window based on its label or model type. Test class instances are created by GUITester for each named component in the window's builder and put into a dictionary. The test writer can then send messages to these test class instances to exercise the widgets. A GUITester Scripting Assistant tool works with TestWorks to make the creation of test scripts very easy. The script can later be modified to include assertions concerning the state of widgets or the underlying domain model, including storing bitmaps of the widgets screen image. A key advantage of this system over other test tools is that the scripting language is Smalltalk itself, so every aspect of the application under test can be queried for state, and engineers do not need to learn a separate language to write tests.