Request for Comments-29: Testing, Benchmarking & Validation of Monteverdi
- 1 Status
- 2 Content
- 2.1 What changes will be made and why they would make a better Orfeo ToolBox?
- 2.2 When will those changes be available (target release or date)?
- 2.3 Who will be developing the proposed changes?
- 3 Community
- 4 Corresponding Requests for Changes
- Author: Stéphane ALBERT
- Submitted on May ??, 2016
- Open for comments
What changes will be made and why they would make a better Orfeo ToolBox?
This Request For Comments is about validating, testing & benchmarking of Monteverdi which is an interactige Graphical User Interface (GUI) software program redering OpenGL scenes.
A classical way to test GUIs is to have human n-users manually testing the software:
- according to some written validation plan (atomic test-cases of functionalities); and
- freely, without any constraints.
This method is:
- requires one or more people who have not been involve in the development/design process
- takes lot of time
- is repetitive (which is source of human errors when validating).
Even if manual validation of GUI applications is hardly avoidable, the goal of this RFC is to discuss ways and tools to automate some parts of the validation, testing and benchmarking of Monteverdi. This will save some production time and increase the overall quality of the software.
To reach this goal, several topics should be addressed:
- How rendering performances could be measured (and eventually compared)?
- Which data should serve as reference testing data?
- How can GUI validation, testing and benchmarking be automated?
- Which production cost will be needed to develop and maintain testings scripts?
- How can validation, testing and benchmarking be integrated into the OTB/Monteverdi release process?
Because Monteverdi cannot be tested with all user data, validation, testing & benchmarking of Monteverdi should be made on some (agreed) representative data and/or combinations of singular items which will be considered as reference data:
- supported formats (TIF, JP2K etc.)
- supported products encoding (PLEIADES, QUICKBIRD, SPOT5 etc.)
- representativily sized data.
- some exceptional test-case data:
- 1x1 pixel images
- 0 representative pixel image such image filled only with no-data value
- Compound Data Set
Rendering performance (basic benchmarking)
Complete benchmarking of GUI, OpenGL rendering plus data IOs could be a complex task. However, an accurate framerate frequency (inverse of rendering time), such as in video games, could be measured using performance counters    and registered into memory buffers for each
otb::GlView and for each frame. When benchmarking/testing is finished, this result could be output into some textual trace file for archiving or non-regression comparison.
Notes about Performance Counters:
- A portable performance counter API should be added into the OTB library and implemented for each supported platform platform (Windows, OSX and GNU/Linux) .
Frame-rate frequencies could be displayed as text in overlay of each
otb::GlView in the Ice engine so that both IceViewer and Monteverdi (and possibly and external application) could benefit of it.
As an option, a multi-color graphical polyline chart of frame-rate frequencies, such a systems process manager, could be displayed as overlay of the
otb::GlView and included in the Ice rendering engine to visually check performances during development and human testing.
Note about benchmarking and performances:
- Benchmarking & performance measurements must be make on some reference platforms (OS, native or virtual-machine, same 3D hardware, same OpenGL driver and libraries) without any other process running on platform except system processes (OS and windowing system) and benchmarking scripts. Otherwise, measurement will not be representative nor comparable.
Automation of Ice rendering engine benchmarking
The Ice rendering engine is used in Monteverdi through Qt OpenGL widgets. The automation of testing these OpenGL widgets (along with their content) is the same as for general GUI widgets (see Automation of GUI testing). However a GUI independant automation of Ice rendering engine benchmarking could be done via the use of an integrated replayer so that rendering benchmarking could be compared from build to build on the same basis.
A replayer API could be implemented into the Ice rendering engine in order to animate move of the view through given arbitrary test data along a pre-defined arbitray polyline path (which simulates user's navigation in the viewport). Several polyline paths could be saved in external file given as input data along with arbitrary input products (see Reference data) and passed to the replayer API. Rendering frequencies could be recorded into memory during the replay and output to textual log files for post-processing after benchmarking has finished. Replay could be run nightly/automatically or at tester's demand with interactive display of rendering frequencies.
Automation of GUI testing
Beside the testing framework provided within the Qt framework, there are several other tools which might be of interest to test the Monteverdi GUI . Some of these tools will be discussed hereafter.
The Qt framework provides the QTest module  within the QTestLib  framework in order to automate testing of GUI components. It can simulate user events by calling C++ functions of the API which activate Qt signal/slots mechanism. Results are then checked via direct C++ function calls . These seem to be usefull tools to automate unit testing of GUI components such as individual widgets. Even if they could be used to test the overall Monteverdi application, it may not be their primary goal, mostly, because they need knowledge of the C++ GUI source code. QTestLib may not be used to test nightly/release packages. However, QTestLib based unig testing may be integrated to the OTB nightly CTest framework and dashboard. This testing framework also provides some benchmarking components.
Note: These tools could also be of good use to test the GUI of OTB-applications.
The QTestLibframework could be usefull to:
- Script unit testing of each Monteverdi component separately
- Script validation of Monteverdi overall GUI by duplicating the Monteverdi
main()into some test class
- Script benchmarking of Monteverdi Ice views
- Include Qt unit testing into CTest nightly testing and dashboard overview
- Optionally, script unit testing of the GUIs of OTB-applications
The QTestLib framework will not be usefull to:
- Test nightly and release packages
The Linux Desktop Testing Project
LDTP  is a cross platform GUI test automation tool which uses the Assistive Technology API (AT-API) to communicate with the desktop and GUI applications provided those are accessibility enabled (which is a feature of the Qt framework and main GNU/Linux desktop environments). This design makes it desktop, GUI API, graphics and source code agnostic. It is based on the concepts of the Software Automation Framework Support .
It provides some scripting languages API (one of which is Python ) to script testing and access the components of the desktop and GUI applications via the AT-API (see extensive tutorial ). Such as for QTestLib the API allows to access GUI widgets, check their content but also, wait for some event or result, conditionnaly check if some dialog has popped up and take screenshots.
Compared to QTestLib, the LDTP grants the same advantages but can also be used to test the nightly and release packages. Disadvantages might be:
- Needs another language to script the tests than C++ (which is used in OTB and Monteverdi)
- Be more difficult to integrate into the CTest unit testing process and dashboard status.
Sikulli  is another tool which is more of an automation scripting facility than a testing tool. It works by using image recognition. So, it may require that the testing scripts be adapted for each desktop theme and/or environment.
The LDTP seems more appropriate than Sikuli.
Development and maintenance of testing scripts
Independently of which testing framework will be chosen, an exhaustive unit testing would require each component and each feature to be unit tested as a single item and also as a part of the whole Monteverdi OpenGL GUI application. If given feature or component can be automatically tested, it would require one or more test classes/functions and/or input data.
Moreover, when the Monteverdi GUI of features are improved or modified, testing scripts would have to be maintained.
Finally, output of unit testing would have to be analyzed regularly.
To be effective, validation, testing and benchmarking of Monteverdi would best be done by being integrated to the release process and run on some frozen and non-evolving source code such as a release or stable candidate or a dedicated stable branch of the source code repository. This point is important because any modification of the source code could invalidate some previously run tests.
For example, a first release-candidate #0 could be done to run the whole validation process to find bugs, regressions, crashes etc. When this step is done, results would be analyzed and registered to the bugtracker and tagged as major, minor, blocking/non-blocking etc. They will then be fixed and a new release-candidate #1 is packaged. Same steps applies until the remaining issues are considered minor and non-blocking.
In this way, we ensure that the validation plan converges to some acceptable target and that the remaining issues are decreasing.
Moreover, even if automatic validation, testings and benchmarking is helpfull, human testers, which are not involved developers nor designers, should be included in the validation process to report bugs and to give their feedback.
When will those changes be available (target release or date)?
Who will be developing the proposed changes?
Corresponding Requests for Changes