Request for Changes-62: Enhance text comparison in dashboard

From OTBWiki
Jump to: navigation, search

[Request for Changes - 62] Enhance text comparison in dashboard

Status

  • Author: Guillaume
  • Additional Contributors (if different than authors)
  • Submitted on 21.09.2016
  • Proposed target release : 5.8
  • Adopted : +4 from Julien, Victor, Rémi, Guillaume
  • Merged : e38b319d14f700f307698437b693bc03822a06e3

Summary

This RFC proposes a new function to compare text files in dashboard tests.

Rationale

The output of the current method comparing text files is not able to detect additional or missing lines. In those cases, the whole comparison is wrong because of a 1-line shift between test and baseline files. In addition, when 2 lines differ, the output doesn't outline the specific parts of the lines that mismatch.

Implementation details

Classes and files

M       Modules/IO/TestKernel/include/otbTestHelper.h
M       Modules/IO/TestKernel/include/otbTestMain.h
M       Modules/IO/TestKernel/src/otbTestHelper.cxx

In the TestHelper, 2 functions have been added : RegressionTestAllDiff() and RegressionTestDiffFile() (in parallel with the existing RegressionTestAllAscii() and RegressionTestAsciiFile()). They perform a comparison between text files like a typical diff program would do, but they have special behaviours for OTB context :

  • Numerical values are compared with a tolerance
  • Some lines can be ignored during comparison
  • Values of memory pointer adresses are ignored during comparison

Internally, the function splits each line into tokens (all the non-alphanumeric characters are used as separators). When 2 lines can be compared, the function shows which token differ.

As an example, here is the comparison of 2 simple files :

  • File base.txt
Sample size: 2
Nb samples : 3
Shifts: [-1, -3]
Scales: [1, 2]
Input samples: 
[-2, -5]
[-1, -3]
[0, -1]
Output samples: 
[-1, -1]
[0, 0]
[1, 1]
  • File test.txt
Sample size: 2
Shifts: [-1, -3]
Scales: [1, 2]
Input samples: 
[-2, -5]
[-1, -3, 0.2]
[0, -1]
TODO
Output samples: 
[-1, -1]
[0, 0]
[1, 1]
  • Output of comparison
<DartMeasurement name="ASCIIFileError" type="numeric/int">3</DartMeasurement>
================================================================
baseline ASCII File : ../base.txt
Test ASCII File     : ../test.txt
Diff ASCII File     : ../test.txt.diff.txt
Tolerance value     : 0
Tolerance max check : 1e-30
Nb lines different : 3
   Base << Nb samples : 3
   -------------------------------
   Base << [-1, -3|
   Test >>  --  --, 0.2]
   -------------------------------
   Test >> TODO
-------------  End control baseline tests    -------------

Applications

Tests

M       Examples/Projections/test/CMakeLists.txt

The numeric tolerance of a test was increased from 10^-30 to 10^-7 to allow a greener dashboard.

The baselines of 2 other tests have been updated as they were obsolete (see branch diff_compare on OTB-Data).

Documentation

Additional notes

Using this new comparison method, 2 tests have been spotted as failing :

  • uaTvPlatformPositionComputeBaselineTest
  • uaTvGeometricSarSensorModelAdapterTest

They are probably linked with developments on TerraSar-X and ossimPlugins.