Rework OTB-Sextante integration

Introduction

• Usefull links :

Main site : http://www.qgis.org
Doc : http://docs.qgis.org/html/en/docs/user_manual/index.html
Doc chapter about Sextante : http://docs.qgis.org/html/en/docs/user_manual/sextante/index.html
C++ API (mostly accessible in Python) : http://qgis.osgeo.org/api
Qgis sources : https://github.com/qgis/Quantum-GIS
Sextante sources : https://github.com/qgis/Quantum-GIS/tree/master/python/plugins/sextante
Redmine instance : http://hub.qgis.org/projects/sextante
Sextante plugin homepage : http://plugins.qgis.org/plugins/sextante/
Sextante issue tracker (has a Category called Backend/OTB) : http://hub.qgis.org/projects/sextante/issues

• Current situation
  • The module model is too different between sextante and OTB
    • OTB has a hierarchical tree of parameters, Sextante params have a flat structure
    • Current implementation just expose all the parameters without restriction, it makes some modules unusable
  • Some modules exposed in the current version of Sextante don't make much sense in the Qgis context and should be filtered out
  • No version handling : the current implementation works only for a specific version of OTB
  • No tests : we should try to reproduce the OTB tests in the Sextante context, reusing the same input and baselines

• Description of the current implementation/workflow
  • txt files describing the different modules are generated in python/plugins/sextante/otb/description
  • they are generated by a developer using the script python/plugins/sextante/otb/helper/generate_application_descriptors.py
    • this script makes use of the python-otb module, importing the module otbApplication, to access all the applications models/descriptions
    • we don't want this python module to be used during sextante execution
  • OTBAlgorithmProvider.py parses the directory containing descriptors
  • for each txt file, it creates a OTBAlgorithm instance, passing the txt file as a constructor parameter
  • OTBAlgorithm.defineCharacteristicsFromFile creates all the Sextante parameters from the txt file
  • OTBAlgorithm.processAlgorithm creates the command line from the sextante parameter value, and executes it.

• Design/Workflow solution
  • a developer generates a set of description files, corresponding to an OTB version, from a script based on python-otb module
  • the description files contains everything needed to describe all the parameters, their doc...
  • some OTB apps are automatically wrapped using an adapted version of the current implementation
  • for others, develop a specific Sextante module :
    • exposing a subset of parameters
    • grabbing all information about the exposed parameters from the description file whenever possible (to get doc, parameter long name, ...)

Applications considered for integration

Feature Extraction

• BinaryMorphologicalOperation

** specific sextante module
** in/out/ram/channel left as-is
** force "-structtype ball", do not expose this param
** expose one "radius" param only. force using same radius for x and y
** expose one foreval/backval only. during command line construction,
   set them all (only the relevant one will be used, the rest will be ignored silently)

• EdgeExtraction

** in/out/ram/channel left as-is
** choice list for filter
** expose one "radius" param only
** doc says the radius is used only for touzi

• GrayScaleMorphologicalOperation

** specific sextante module
** in/out/ram/channel/filter left as-is
** force "-structtype ball", do not expose this param
** expose one "radius" param only. force using same radius for x and y

• HaralickTextureExtraction

** automatic wrapping should work, expose all params

• LocalStatisticsExtraction

** automatic wrapping should work, expose all params

• RadiometricIndices

** the otb app can compute several indices at once, but the StringList parameter is unmanageable
** transform the otb "StringList" to a Sextante "choice" : the sextante app will only compute one indice
** rest of parameters unchanged

• SFSTextureExtraction

** automatic wrapping should work, expose all params

• LineSegmentDetection

** automatic wrapping should work, expose all params

Change Detection

• MultivariateAlterationDetector

** automatic wrapping should work, expose all params

Dimensionnality Reduction

• DimensionnalityReduction

** create one sextante module for each method : PCA, NA-PCA, MAF, ICA
** expose "outinv" except for MAF

Geometry

• Pansharpening

** create one sextante module for each method : RCS, LMVM, Bayesian
** only Bayesian has dependent parameters

• Superimpose

** automatic wrapping should work, expose all params

• BundleToPerfectSensor

** TBD
** do we want this one ?
** make a sextante "script" module based on Superimpose + Pansharpening ?

• RigidTransformResample

** one module for each transformation type (id="change spacing", translation, rotation)

• HomologousPointsExtraction

** automatic wrapping should work, expose all params
** mode.geobins.binsize & mode.geobins.binstep are the only params dependent on another one.
   ensure the doc or param name reflect that.

• Orthorectification

** TBD

• GenerateRPCSensorModel

** expose all params, except the "map".
** force the use of "-map wgs"

• RefineSensorModel

** expose all params, except the "map".
** force the use of "-map wgs"

Segmentation

• Segmentation

** one module for each segmentation algorithm
** skip "edison meanshift" algo
** force "-mode vector". do not consider the mode "raster" (Standard segmentation with labeled output in the otbgui_)
** Issue : the params "-mode.vector.minsize" and "-mode.vector.simplify" are numeric and non-mandatory (off by default). this does not exist in Sextante. we need to associate a boolean to use them : a boolean "Simplify polygons" + a numeric parameter "Simplification radius". same goes for "minsize".

Classification

• KMeansClassification

** automatic wrapping should work, expose all params
** do not expose the "rand" param

• ClassificationMapRegularization

** automatic wrapping should work, expose all params

• FusionOfClassifications

** automatic wrapping should work, expose all params

• ComputeImagesStatistics

** automatic wrapping should work, expose all params

• TrainSVMClassifier

** automatic wrapping should work
** remove params : "-rand"

• ImageSVMClassifier

** automatic wrapping should work, expose all params

• ComputeConfusionMatrix

** expose the mode "ref=vector" only for a starter...

Radiometry

• OpticalCalibration

** one module for TOA calibration
 *** in/out/ram/level/milli/noclamp
** another module for TOC calibration
 *** all parameters

• SarRadiometricCalibration

** before doing anything, check support for SAR data in Qgis
** automatic wrapping should work, expose all params

Utils

• BandMath

** TBD : duplicates a functionnality already available at several places (specific plugins, some sextante modules)
** do other implementation support streaming ?

• ConcatenateImages

** automatic wrapping should work, expose all params

• Convert

** TBD. Call it RGBRenderer ?

• Rescale

** TBD. Duplicates Convert in some ways.

• Smoothing

** one module with "mean" and "gaussian"
** another specific module with "anisotropic" mode

• SplitImage

** Rename "split bands"
** expose all params

• ReadImageInfo

** expose at least a "geom extractor module"
** in & outkwl params only

• ImageEnveloppe

** expose all params
** need to use Qgis projection settings to retrieve a WKT code

Stereo

• StereoFramework

** expose all parameters

Full list of applications

This is the list of applications available in OTB 3.16.

Legend is :

---- : already handled. Sextante wrapping is exposed in the previous paragraph

strike text : the OTB application is not exposed in Sextante

none of the above : Give your opinion !

• ---- BandMath
• ---- BinaryMorphologicalOperation
• BlockMatching : for now, we just wraps the "allinone" stereo app
• ---- BundleToPerfectSensor
• ---- ClassificationMapRegularization
• ColorMapping
• CompareImages
• ---- ComputeConfusionMatrix
• ---- ComputeImagesStatistics
• ComputePolylineFeatureFromImage : too specific appli. not applicable in Qgis
• ---- ConcatenateImages
• ConcatenateVectorData : surely there are better ways to do that in Qgis
• ConnectedComponentSegmentation : duplicate of Segmentation appli
• ---- Convert
• ConvertCartoToGeoPoint : too specific appli. not applicable in Qgis
• ConvertSensorToGeoPoint : too specific appli. not applicable in Qgis
• DSFuzzyModelEstimation : too specific appli. not applicable in Qgis
• ---- DimensionalityReduction
• DisparityMapToElevationMap : for now, we just wraps the "allinone" stereo app
• ---- EdgeExtraction
• ExtractROI : clone of gdal_translate (Translate or Clipper gdal tool in Qgis)
• FineRegistration : usually does not converge with big rasters
• ---- FusionOfClassifications
• ---- GenerateRPCSensorModel
• ---- GrayScaleMorphologicalOperation
• GridBasedImageResampling : for now, we just wraps the "allinone" stereo app
• ---- HaralickTextureExtraction
• ---- HomologousPointsExtraction
• HooverCompareSegmentation : too specific appli. not applicable in Qgis
• HyperspectralUnmixing
• ---- ImageEnvelope
• ---- ImageSVMClassifier
• ---- KMeansClassification
• KmzExport
• ---- LineSegmentDetection
• ---- LocalStatisticExtraction
• MaximumAutocorrelationFactor : duplicate of DimensionnalityReduction
• MeanShiftSmoothing : duplicated in Segmentation
• MultiResolutionPyramid : gdaladdo clone
• ---- MultivariateAlterationDetector
• OSMDownloader : supported by GDAL since recently. See here
• ObtainUTMZoneFromGeoPoint : too specific
• ---- OpticalCalibration
• ---- OrthoRectification
• ---- Pansharpening
• PixelValue : too specific appli. not applicable in Qgis
• Quicklook : duplicate of gdalwrap
• ---- RadiometricIndices
• Rasterization : duplicate of gdal_rasterize (just adds the sensor geometry handling)
• ---- ReadImageInfo
• ---- RefineSensorModel
• ---- Rescale
• ---- RigidTransformResample
• ---- SFSTextureExtraction
• SOMClassification
• ---- SarRadiometricCalibration
• ---- Segmentation
• ---- Smoothing
• ---- SplitImage
• ---- StereoFramework
• StereoRectificationGridGenerator : for now, we just wraps the "allinone" app
• StereoSensorModelToElevationMap : for now, we just wraps the "allinone" app
• ---- Superimpose
• TestApplication : no need for this one
• TileFusion
• ---- TrainSVMImagesClassifier
• ValidateSVMImagesClassifier : duplicate the functionnality of ConfusionMatrix computation ?
• VectorDataDSValidation : too specific
• VectorDataExtractROIApplication : Qgis is better than OTB for that
• VectorDataReprojection : Qgis is better than OTB for that
• VectorDataSetField : Qgis is better than OTB for that
• VectorDataTransform : Qgis is better than OTB for that
• VertexComponentAnalysis