View source for Request for Comments-29: Testing, Benchmarking & Validation of Monteverdi

= Status =
* Author: Stéphane ALBERT
* Submitted on May ??, 2016
* Open for comments

= Content =

== What changes will be made and why they would make a better Orfeo ToolBox? ==

This Request For Comments is about validating, testing & benchmarking of <i>Monteverdi</i> which is an interactige Graphical User Interface (GUI) software program redering <i>OpenGL</i> scenes.

A classical way to test GUIs is to have human n-users manually testing the software:
* according to some written validation plan (atomic test-cases of functionalities); and
* freely, without any constraints.

This method is:
* lengthy
* requires one or more people who have not been involve in the development/design process
* takes lot of time
* is repetitive (which is source of human errors when validating).

Even if manual validation of GUI applications is hardly avoidable, the goal of this RFC is to discuss ways and tools to automate some parts of the validation, testing and benchmarking of <i>Monteverdi</i>. This will save some production time and increase the overall quality of the software.

To reach this goal, several topics should be addressed:
# How rendering performances could be measured (and eventually compared)?
# Which data should serve as reference testing data?
# How can GUI validation, testing and benchmarking be automated?
# Which production cost will be needed to develop and maintain testings scripts?
# How can validation, testing and benchmarking be integrated into the <i>OTB</i>/<i>Monteverdi</i> release process?

==== Data ====

Because <i>Monteverdi</i> cannot be tested with all user data, validation, testing & benchmarking of Monteverdi should be made on some (agreed) representative data and/or combinations of singular items which will be considered as reference data:
* supported formats (TIF, JP2K etc.)
* supported products encoding (PLEIADES, QUICKBIRD, SPOT5 etc.)
* representativily sized data.
* some exceptional test-case data:
** 1x1 pixel images
** 0 representative pixel image such image filled only with no-data value
* Compound Data Set
* etc.

==== Rendering performance (basic benchmarking) ====

Complete benchmarking of GUI, <i>OpenGL</i> rendering plus data IOs could be a complex task. However, an accurate framerate frequency (inverse of rendering time), such as in video games, could be measured using performance counters [http://msdn.microsoft.com/en-us/library/windows/desktop/ms644905%28v=vs.85%29.aspx] [http://msdn.microsoft.com/en-us/library/windows/desktop/ms644904%28v=vs.85%29.aspx] [http://stackoverflow.com/questions/12468331/queryperformancecounter-linux-equivalent] and registered into memory buffers for each <code>otb::GlView</code> and for each frame. When benchmarking/testing is finished, this result could be output into some textual trace file for archiving or non-regression comparison.

<u>Notes about Performance Counters</u>:
: A portable performance counter API should be added into the OTB library and implemented for each supported platform platform (Windows, OSX and GNU/Linux) [http://tdistler.com/2010/06/27/high-performance-timing-on-linux-windows].

Moreover, the <code>otb::GlView::HeaveRender()</code> [http://git.orfeo-toolbox.org/otb.git/blob/refs/heads/develop:/Modules/Visualization/Ice/src/otbGlView.cxx#l264] performs data update before calling <code>::LightRender()</code> [http://git.orfeo-toolbox.org/otb.git/blob/refs/heads/develop:/Modules/Visualization/Ice/src/otbGlView.cxx#l218]. So, <code>::HeavyRender()</code> time equals <code>::LightRender()</code> plus data update time.

Frame-rate frequencies could be displayed as text in overlay of each <code>otb::GlView</code> in the <i>Ice</i> engine so that both <i>IceViewer</i> and <i>Monteverdi</i> (and possibly and external application) could benefit of it.

As an option, a multi-color graphical polyline chart of frame-rate frequencies, such a systems process manager, could be displayed as overlay of the <code>otb::GlView</code> and included in the <i>Ice</i> rendering engine to visually check performances during development and human testing.

<u>Note about benchmarking and performances</u>:
: Benchmarking & performance measurements must be make on some reference platforms (OS, native or virtual-machine, same 3D hardware, same OpenGL driver and libraries) without any other process running on platform except system processes (OS and windowing system) and benchmarking scripts. Otherwise, measurement will not be representative nor comparable.

==== Automation of <i>Ice</i> rendering engine benchmarking ====

The <i>Ice</i> rendering engine is used in <i>Monteverdi</i> through <i>Qt</i> <i>OpenGL</i> widgets. The automation of testing these <i>OpenGL</i> widgets (along with their content) is the same as for general GUI widgets (see Automation of GUI testing). However a GUI independant automation of <i>Ice</i> rendering engine benchmarking could be done via the use of an integrated replayer so that rendering benchmarking could be compared from build to build on the same basis.

A replayer API could be implemented into the <i>Ice</i> rendering engine in order to animate move of the view through given arbitrary test data along a pre-defined arbitray polyline path (which simulates user's navigation in the viewport). Several polyline paths could be saved in external file given as input data along with arbitrary input products (see Reference data) and passed to the replayer API. Rendering frequencies could be recorded into memory during the replay and output to textual log files for post-processing after benchmarking has finished. Replay could be run nightly/automatically or at tester's demand with interactive display of rendering frequencies.

==== Automation of GUI testing ====

Beside the testing framework provided within the <i>Qt</i> framework, there are several other tools which might be of interest to test the <i>Monteverdi</i> GUI [http://doc.qt.io/qt-4.8/qttest-module.html#qttest]. Some of these tools will be discussed hereafter.

===== QTestLib =====

The <i>Qt</i> framework provides the <i>QTest</i> module [http://doc.qt.io/qt-4.8/qttest-module.html#qttest] within the <i>QTestLib</i> [http://doc.qt.io/qt-4.8/qtestlib-manual.html] framework in order to automate testing of GUI components. It can simulate user events by calling C++ functions of the API which activate Qt signal/slots mechanism. Results are then checked via direct C++ function calls [http://doc.qt.io/qt-4.8/qtestlib-tutorial.html]. These seem to be usefull tools to automate unit testing of GUI components such as individual widgets. Even if they could be used to test the overall <i>Monteverdi</i> application, it may not be their primary goal, mostly, because they need knowledge of the C++ GUI source code. <i>QTestLib</i> may not be used to test nightly/release packages. However, <i>QTestLib</i> based unig testing may be integrated to the OTB nightly <i>CTest</i> framework and dashboard. This testing framework also provides some benchmarking components.

<u>Note</u>: These tools could also be of good use to test the GUI of OTB-applications.

The <i>QTestLib</i>framework could be usefull to:
* Script unit testing of each <i>Monteverdi</i> component separately
* Script validation of <i>Monteverdi</i> overall GUI by duplicating the <i>Monteverdi</i> <code>main()</code> into some test class
* Script benchmarking of <i>Monteverdi</i> <i>Ice</i> views
* Include Qt unit testing into <i>CTest</i> nightly testing and dashboard overview
* Optionally, script unit testing of the GUIs of OTB-applications

The <i>QTestLib</i> framework will not be usefull to:
* Test nightly and release packages

===== The Linux Desktop Testing Project =====

<i>LDTP</i> [http://ldtp.freedesktop.org/wiki] is a cross platform GUI test automation tool which uses the <i>Assistive Technology</i> API (<i>AT-API</i>) to communicate with the desktop and GUI applications provided those are accessibility enabled (which is a feature of the <i>Qt</i> framework and main GNU/Linux desktop environments). This design makes it desktop, GUI API, graphics and source code agnostic. It is based on the concepts of the <i>Software Automation Framework Support</i> [http://safsdev.sourceforge.net/Default.htm].

It provides some scripting languages API (one of which is Python [http://ldtp.freedesktop.org/user-doc/index.html]) to script testing and access the components of the desktop and GUI applications via the AT-API (see extensive tutorial [http://ldtp.freedesktop.org/ldtp-tutorial.pdf]). Such as for <i>QTestLib</i> the API allows to access GUI widgets, check their content but also, wait for some event or result, conditionnaly check if some dialog has popped up and take screenshots.

Compared to <i>QTestLib</i>, the <i>LDTP</i> grants the same advantages but can also be used to test the nightly and release packages. Disadvantages might be:
* Needs another language to script the tests than C++ (which is used in <i>OTB</i> and <i>Monteverdi</i>)
* Be more difficult to integrate into the <i>CTest</i> unit testing process and dashboard status.

===== Sikulli =====

<i>Sikulli</i> [http://www.sikuli.org] is another tool which is more of an automation scripting facility than a testing tool. It works by using image recognition. So, it may require that the testing scripts be adapted for each desktop theme and/or environment.

The <i>LDTP</i> seems more appropriate than <i>Sikuli</i>.

==== Development and maintenance of testing scripts ====

Independently of which testing framework will be chosen, an exhaustive unit testing would require each component and each feature to be unit tested as a single item and also as a part of the whole <i>Monteverdi</i> <i>OpenGL</i> GUI application. If given feature or component can be automatically tested, it would require one or more test classes/functions and/or input data.

Moreover, when the <i>Monteverdi</i> GUI of features are improved or modified, testing scripts would have to be maintained.

Finally, output of unit testing would have to be analyzed regularly.  

==== Validation process ====

To be effective, validation, testing and benchmarking of <i>Monteverdi</i> would best be done by being integrated to the release process and run on some <b>frozen and non-evolving source code</b> such as a release or stable candidate or a dedicated stable branch of the source code repository. This point is important because any modification of the source code could invalidate some previously run tests.

For example, a first release-candidate #0 could be done to run the whole validation process to find bugs, regressions, crashes etc. When this step is done, results would be analyzed and registered to the bugtracker and tagged as major, minor, blocking/non-blocking etc. They will then be fixed and a new release-candidate #1 is packaged. Same steps applies until the remaining issues are considered minor and non-blocking.

In this way, we ensure that the validation plan converges to some acceptable target and that the remaining issues are decreasing.

Moreover, even if automatic validation, testings and benchmarking is helpfull, human testers, which are not involved developers nor designers, should be included in the validation process to report bugs and to give their feedback.

== When will those changes be available (target release or date)? ==

OTB-5.6/Monteverdi-3.4

== Who will be developing the proposed changes? ==

TBD

= Community =

=== Comments ===

=== Support ===

= Corresponding Requests for Changes =

TODO