GSoC 2014 project

From OTBWiki
Jump to: navigation, search
Student Name: Martina Porfiri, University of Rome "La Sapienza"
Organization: OSGeo - Open Source Geospatial Foundation
OSGeo Foundation member: OTB - Orfeo Toolbox
Mentors Name: Manuel Grizonnet, Conrad Bielski, Ned Horning
Project Title: OTB ICE extension for Monteverdi2: implementation of dynamic functionalities for satellite image time series analysis
Repository: Mercurial-Bitbucket


Contents

Introduction

The Earth is a dynamic system and its surface is continuously changing at different time scales. Accordingly, Satellite Image Time Series (SITS) are an essential resource to analyze and monitor the Earth surface dynamics. In the coming years, both high temporal and high spatial resolution SITS are going to be widely and freely available thanks to the European Space Agency’s (ESA) Sentinel program. In order to efficiently use the huge amounts of data that will be produced by, for instance, Sentinel-2 (global cover every five days with 10 m to 60 m resolution and 13 spectral bands), the goal of this project is to add new OTB Monteverdi2 functionalities for SITS visualization, based on ICE and OpenGL APIs. Hereafter is reported a draft proposal (tasks and related time schedule) that will be tuned on the way, in accordance with the mentoring organization.

Background

Recently, ORFEO ToolBox has expanded its open source library with a new and fast OpenGL rendering API, called ICE, for remote sensing images. Currently, basic and very useful functionalities (e.g. zooming and panning, different portals and shaders, multiple images managing whatever the sensor or the geometry) have been implemented. Due to the incoming ESA Sentinel satellite missions, huge time series data will be widely available and new image visualization GUIs need to be added in order to manage efficiently all derived layers in Monteverdi2. Good functionalities could be ‘cover flow’, that allows to view all loaded images in a dynamic and efficient way, and ‘criteria ordering’, that allows to classify the data depending on different characteristics (e.g. time, spectral band, sensor). Processing image time series means processing huge amounts of data acquired not only in different time, but also with different geometric configuration and spatial resolution in several spectral bands (up to 13 spectral bands for Sentinel-2). Therefore, it could be useful to develop and implement several visualization tools that enable to manage the data depending on both time and spectral band.

The idea

The idea is to develop new functionalities for OTB ICE extension for satellite image time series visualization in Monteverdi2. The project will be based on OpenGL API and it will be implemented in C++ language. I am interested to investigate the following main aspect:

  • implementation of new and specific functionalities, as cover flow mode and criteria ordering definition, able to easily browse and search dozens of images derived from satellite time series

In particular, my idea is to supply to the users the following procedures:

  • possibility to view, browse and search all loaded data using the cover flow user interface, available in OpenGL in different kind of viewing. The idea is to create a panel type ‘Multi Image GUI’ similar to the actual Monteverdi2 one, where the users will be able to visualize and select the data in two different cover flow mode, i.e. default (classical cover flow stack representation) or flat (horizontal stack visualisation). Flipping through snapshots of images, the main metadata information (eventually selectable by the users) and the main characteristics of the selected image will be showed, respectively, in the actual docking windows ‘Dataset properties’, ‘Quicklook view’, ‘Histogram’ and ‘Pixel Description’. Moreover, the data list will be reported in the ‘Name’ window, from which will be possible to select the image of interest. Like this, one could switch the data, visualize the relative selected information and select the images to process in a dynamic and efficient way.
  • possibility to order the data depending on different characteristics specially designed for multispectral satellite sensors (e.g. Sentinel-2, WorldView-1 and -2, RapidEye, Pleiades) as time, spectral band or geometric configuration. This function enables to manage the data depending on the aim of the future processing (e.g. change detection, feature extraction, hyperspectral analysis).
  • possibility to select a stack of images filtering loaded data on specific metadata information of interest, as acquisition date, incidence angle, spectral band, estimated ground spacing. Different ‘Multi Image GUI’ panels will be created: one for all data (e.g. named ‘All Datasets’), one for selected data (e.g. named ‘Selected Datasets’).

In summary, in Monteverdi2 the following steps could be performed:

  • loading all available data, acquired in different time and in different geometric configuration, in ‘All Datasets’ panel
  • filtering loaded data depending on the specific processing application and their visualisation on ‘Selected Datasets’ panel
  • visualizing all and filtered data in coverflow mode (default or flat) and ordering the images depending on specific characteristics

In my opinion, in this way one could manage efficiently the huge number of layers derived from image time series analysis.

Project timeline (reviewed after mid-term)

April 21 - May 18 (Before the official coding time):

  • Familiarizing with OTB ICE library development
  • Refining project plan according to mentor(s) suggestions
  • Preparing wiki page and mercurial repository for code regular hosting

May 19 (Official coding period starts) - June 01:

  • OpenGL with 2D and 3D Graphics analysis
  • Starting ‘GlCoverFlow’ class implementation
  • Testing and controlling implementation efficacy - discussing results with mentor(s)

June 02 - June 22:

  • Continuing to implement ‘GlCoverFlow’ class
  • Testing and controlling implementation efficacy - discussing results with mentor(s)
  • Buffering period

June 23 (Mid-Term) – June 29:

  • Bugs fixing - report adjusting in accordance with mentor(s) suggestions - finalize mid-term report

June 30 - July 20:

  • Refining ‘GlCoverFlow’ class
  • Starting to develop the ‘CoverFlow’ application based on OTB-ICE APIs
  • Testing and controlling implementation efficacy - discussing results with mentor(s)

July 21 - August 03:

  • Continuing to implement ‘CoverFlow’ application
  • Testing and controlling implementation efficacy - discussing results with mentor(s)

August 04 – August 10

  • Testing the ‘CoverFlow’ application controls and interaction with satellite image time series (using available dataset)
  • Discussing results with mentor(s)

August 11- August 18 (pencils down):

  • Buffering period
  • Testing and debugging code - exchanging opinions with mentor(s)
  • Refining documentation and tutorials

After August 18

  • Working on further bugs, dependencies (if any arise)
  • Remaining an active member of the OTB community


DELIVERABLES

  • Source Code of the extension
  • Documentation

About the code repository

On Mercurial-Bitbucket repository are available the following folders:

Dependencies

In order to be able to run the applications, it is necessary, at first, clone and compile OTB and OTB-ICE source using CMake

Summary of set up

To perform a 3D rendering of the loaded images (i.e. allowing their movement in x, y and -especially- z directions on OpenGL screen) it is necessary to rebuild all OTB-ICE source code adding all files reported in 'Code' and 'Application' folders of this repository.

Configuration

  • add 'Code->otbGlCoverFlowView' class in your 'OTB-ICE_source_directory->Code' folder
  • replace 'Code->otbGlView.h' file patch with that one in your 'OTB-ICE_source_directory->Code' folder (in this new file 'otbGlCoverFlowView' class is declared as friend of 'otbGlView' in order to inherit its private members)
  • add all 'Application' files in your 'OTB-ICE_source_directory->Application' folder
  • download the images (hereafter named img1, img2, ..., imgN) in 'Data->Image.tif' or in 'Data->Image.jpg'
  • run 'otbice_CoverFlow img1_path img2_path ... imgN_path' to have a dynamic CoverFlow implemented using in very limited parts ICE APIs (here images are loaded using SOIL library): pressing LEFT and RIGHT KEYS one can see loaded images move in CoverFlow mode ('WARNING': using SOIL, this application allows to menage only the following image formats: BMP (load & save), TGA (load & save), DDS (load & save), PNG (load), JPG (load))
  • run 'otbglcoverflow img1_path img2_path ... imgN_path' to have a dynamic CoverFlow implemented using exclusively ICE APIs (both for loading and rendering images): pressing LEFT and RIGHT KEYS one can see loaded images move in CoverFlow mode ('WARNING': at the moment geotiff files are not supported)

Weekly reports

Community bonding period

Work done

  • I have created current Wiki page for the project and the weekly reports upload
  • I have created a BITBUCKET account where doing a new repository and sharing the code, available at [1]
  • I have got a phone call and several email exchanges with my mentors in order to discuss project refinement and development
  • I have cloned and built OTB, OTB-ICE, Monteverdi2 sources using Ubuntu 12.04
  • I have tried to open images (.JPG, .JP2, .TIFF) using OTB-ICE viewer. I get the problems listed in the ‘Blocking issues’ section
  • I have rethought to the project on the basis of tools interaction and mentors suggestions

Work to be done in the next week

  • I would like to fix OTB-ICE and Monteverdi2 viewer problems
  • I will begin to analyze more in details OTB-ICE code
  • I will try to compile OTB, OTB-ICE and Monteverdi2 also on Mac OS X 10.9.2

Blocking issues

I get some errors trying to open an TIFF images using OTB-ICE viewer. Consequently, I have problems opening also Monteverdi2 viewer.

Week 1: 19 May 2014

Work done

  • Thanks to otb-developer community and mentors suggestions and support, I have fixed (on Ubuntu 12.04) OTB-ICE and Monteverdi2 viewer issues found during community bonding period
  • I got more familiar with OTB-ICE code

Work to be done in the next week

  • I will analyse OpenGL with 2D Graphics
  • I will start 'otbGlCoverFlow' class implementation for OTB-ICE CoverFlow multi image visualization
  • I would like to fix Mac OS X compilation issues

Blocking issues

Trying to compile OTB on Mac OS X I still get some issues that I am analysing with mentors

Week 2: 26 May 2014

Work done

  • I got more familiar with OpenGL with 2D and 3D Graphics
  • I have created an Eclipse project on Ubuntu 12.04
  • I have analized in more details the ’otbIceViewer’ class doing several tests in order to understand better its dependencies and how integrate my new ‘otbGlCoverFlow’ class in it
  • I have added in ‘otbIceViewer.cxx’ file a specific ‘Non-static callbacks’ in order to activate ‘CoverFlow multi images visualization’ from keyboard when the executable ‘otbIceviewer’ is running
  • I have implemented first functions in ‘otbGlCoveFlow’ class, as:
    • ‘Act_Pos’ to compute the position of the current actor (i.e. the selected image)
    • ’Act_Left’ and ‘Act_Right’ to move the actors (images) respectively on the left and on the right direction

Work to be done in the next week

  • I will continue to study ‘otbIceViewer’ class and relative dependencies
  • I will continue the ‘otbGlCoverFlow’ class implementation and its integration in ‘otbIceViewer’ class

Blocking issues

  • This week I focused my attention mainly on the coding, thus I did not fix the Mac OS X compilation issue (but at the moment this is not a real ‘Blocking issue’)

Week 3: 02 June 2014

Work done

  • In ‘otbIceViewer’ I have tested the ‘Non-static callbacks’ created for ‘CoverFlow multi images visualization’ enabling it to move the open image on the left direction of a precise value (=2*deltax)
  • In ‘otbGlCoveFlow’ class I have implemented ‘Act_Animate’ function in order to compute and apply positions to the actors (images)
  • I have fixed OTB and Ice compilation issue on Mac OS X

Work to be done in the next week

  • I will continue the ‘otbGlCoverFlow’ class implementation and its integration in ‘otbIceViewer’ class

Blocking issues

  • No

Week 4: 09 June 2014

Work done

  • I have arranged 'Act_Transf' function in order to compute actor (image) transformation -translation and rotation- using OpenGL matrix functions, as glTranslatef and glRotatef; I have also implemented other main CoverFlow class functions as:
    • 'CF_Des' to design (construct) the CoverFlow animation
    • 'Anim_stop' to stop the CoverFlow animation when required
    • 'CF_Update' to update the CoverFlow animation
    • 'Act_Remove' to remove an actor (image) from the loaded dataset
  • I have analyzed how linking CoverFlow class to otbIceviewer, in order to visualise and test the specific visualization functionalities
  • I have pushed preliminary code to the remote repository. At the moment the code is thought as a patch to be added at the main OTB-Ice repository (it is not an independent application). For more information about it, see the repository site or wikipage (find the links at the begin of this page)
  • According to the mentors suggestions, I began to build tests using Ctest tool

Work to be done in the next week

  • Discussing with the mentors, I will refine the CoverFlow class functions implementation (in particular 'Act_Transf') and relative link to otbIceViewer
  • I will finalize Ctest configuration

Blocking issues

  • At the moment I am slightly slowed down by understanding how efficiently link CoverFlow-otbIceViewer classes

Week 5: 16 June 2014

Work done

  • I have implemented 'Act_Transf' function in order to apply the traslation and rotation matrix to the loaded actors (images), enabling two different modes:
    • images are visualised without view angle rotation
    • images are visualised with view angle rotation
  • I continued to analyze how linking CoverFlow class to otbIceviewer
  • I have refined the CoverFlow class and I have updated the remote repository
  • I have created a cDashboard (available at cDash) in order to execute the tests and to log their results

Work to be done in the next week

  • I will continue to study the 'classes linking' issue
  • I will continue to configure project code tests using CTest

Blocking issues

  • This week I was mainly focused on the 'classes linking' issue. I did not solve it completely, but I guess that, also thanks to mentors support, the key is near

Week 6: 23 June 2014

Work done

  • I have linked OpenGL transformation matrices to loaded actor (image), achieving in glActorImage class; at the moment I am able to apply partial translation and rotation just in the x and y direction (2D)
  • I have investigated the usage and the application of OpenGL functions for managing loaded image also in the 3D space, as:
    • 'gluPerspective', to set up a perspective projection matrix
    • 'glMatrixMode', to specify which matrix is the current matrix; I focused my attention on GL_MODELVIEW and GL_PROJECTION modes

Work to be done in the next week

  • I will analyze how applying right translation and rotation matrices also in the z direction (3D), implementing specific method in GlImageActor class (and other OTB-ICE classes if it will be necessary)
  • I will resume project tests building

Blocking issues

  • Even if I understood how applying transformation matrices on loaded image, at the moment I am unable to move it how I would (i.e. in the 3D space and without distortions) and I have to better investigate and understand OpenGL functions usage and potentialities

Week 7: 30 June 2014

Work done

  • Exploiting OpenGL functions, I have implemented the binary file code in order to test the GlCoverFlow class.
Status: running it one can see loaded images moving on the left and on the right; refinements are required
  • Discussing with mentors and following their hints, I have reviewed the project timeline (available at the wiki page link), giving more value to the integration of GlCoverFlow class as a dedicated application based on OTB-ICE APIs
  • I have pushed new and updated files to the remote repository

Work to be done in the next week

  • I will refine the code to generate the CoverFlow binary code
  • I will start to look how develop the ‘CoverFlow’ application based on OTB-ICE APIs, relying on work done as far

Blocking issues

  • This week I continued to analyze how integrating GlCoverFlow class in otbiceviewer application; this involved a deeply classes integration in ICE, that actually is not the principal aim of this project and, moreover, was slowing me too. For these reasons, with the mentors support, I decided to slightly modify my operating and to start the development of a different application, completely dedicated to CoverFlow

Week 8: 07 July 2014

Work done

  • I have refined the binary file code in order to test the GlCoverFlow class
Status: running it one can see loaded images moving on the left and on the right in CoverFlow mode
  • I learnt more about Model-View-Controller pattern for implementing user interfaces
  • I have implemented and tested a specific application (named otbice_CoverFlow and executable from ICE) in order to visualize loaded images in CoverFlow mode using ICE APIs
  • I have started to analyse how exploit ICE APIs for otbice_CoverFlow application, focusing my attention on GlImageActor ICE class for the images loading

Work to be done in the next week

  • I will continue to study how use ICE APIs for application rendering

Blocking issues

  • No

Week 9: 14 July 2014

Work done

  • I have integrated otbice_CoverFlow application with GetExtent GlImageActor function, in order to retrieve the image dimensions
  • I have refined otbice_CoverFlow application usage, i.e. now the images can be loaded directly form the terminal shell
  • I have implemented a new application in order to focus my attention mainly on the integration issue, rendering a single image in the most simple way (just load and visualize it) using only ICE APIs

Work to be done in the next week

  • I will discuss with mentors how fixing the issue regarding ICE APIs and otbice_CoverFlow integration. Depending on it, I will refine the rendering application and the GlCoverFlow class

Blocking issues

  • This week I get some problems in understanding how using some GlImageActor functions in order to load and render images. BTW I am discussing with mentor a solution

Week 10: 21 July 2014

Work done

This week my work was focused on two parallel issues regarding ICE code usage: i) APIs+Viewer integration and ii) 3D image rendering. In order to solve them I have implemented a new simple Viewer, 'basicviewer': at the moment through this application the user is able to load and visualize a single image in a three dimensional view (doing it efficiently have required some days). Hereafter are pointed out the two steps:

  • Implementing and testing the new application, I have better understood how correctly integrate ICE APIs, in particular 'GlImageActor' and 'GlView' class ones, for image rendering
  • I have investigated how implementing 3D rendering, i.e. how moving loaded images also in the z-direction in order to allow the CoverFlow mode. It was necessary modifying the 'orb::GlView::BeforeRendering' function: in particular switching from 'glOrtho' projection to 'gluPerspective' the 3D moving has been enabled. In this context a crucial role was played by the function parameters: wrong ones push the image outside the view frustum and thus make it not visible on screen

Work to be done in the next week

  • I will restart to develop the 'otbice_CoverFlow' application aimed to CoverFlow images rendering using ICE APIs. Most probably it will be necessary reorganize and adjust 'GlCoverFlow' class structure

Blocking issues

  • Even if it was not trivial, this week I fixed my blocking issues

Week 11: 28 July 2014

Work done

In order to better explain work done during this week, it could be useful do a brief summary. At first I was focused on restarting the implementation of otbice_CoverFlow application using ICE APIs; in order to not modify the initial ICE code, it was necessary to implement a new class (GlView_CF) to manage the rendering. Because it get some rendering problems, I chose to left this application and resume basicviewer one. Find hereafter a more detailed report:

  • I have restarted the implementation of otbice_CoverFlow in order to integrate ICE APIs. This has required:
    • the implemention of a new viewing class (GlView_CF) that, inheriting GlView one and having its specific rendering functions (as BeforeRendering_CF, LightRender_CF, HeavyRender_CF), allows to manage image rendering without modify the initial ICE code
    • the reorganization of the GlCoverFlow class, i.e. replacing rendering functions with ICE ones
  • Because the previous way turned out to be not efficient, I chose a more linear way. Doing a backward step, I have implemented image rendering using basicviewer application and GlView_CF class. It was also necessary to update basicviewer structure for loading more than one images. At the moment, basicviewer application allows to load three images in a ‘static CoverFlow’: the centered image (second loaded) is in evidence respect the others

Static CoverFlow.png

Work to be done in the next week

  • I will continue to implement basicviewer for image rendering computing automatically relative positions between images (now positions are set one by one). The idea is to implement the CoverFlow module directly in GlCover_CF in order to simplify the procedure
  • I will investigate more details OpenGL rendering function in order to create an efficient view perspective for the CoverFlow visualization

Blocking issues

  • This week I obtained a first result, but I still have to better investigate rendering functions

Week 12: 04 August 2014

Work done

  • I have continued to implement and refine GlCoverFlowView class (ex GlView_CF) in order to switch from a static to a dynamic CoverFlow mode, running the otbglcoverflow (ex basicviewer) application
  • Following mentors suggestions, I was focused on two issues:
    • enabling to load and manage a generic number of images (not just three like before)
    • activating images movement when pressing specific keys (left and right)

In order to perform this I have implemented four new functions in GlCoverFlowView class, in particular:

  1. 'First_Pos':where the image positions are initialized
  2. 'MoveLeft':where the image to be rendered in evidence is selected when the left key is pressed
  3. 'MoveRight':where the image to be rendered in evidence is selected when the right key is pressed
  4. 'UpdatePos':where the image positions are updated when the left or right keys are pressed
  • In this way, running otbglcoverflow application one can load n images and, pressing left and right keys, see them (just a part) moving in a 'coarse' CoverFlow mode. At the moment there are two main limitations (see a brief demonstration at CoverFlow):
    • pressing keys beyond the last (or first) image stops the movement
    • images do not page through along the x direction, thus images loaded beyond screen edges are not visible

Work to be done in the next week

  • I will refine GlCoverFlowView class in order to pass actual limitations and obtain a more efficient CoverFlow
  • I will test the otbglcoverflow application controls and interaction with several images (using available dataset)​, discussing results with mentors​
  • I will generate and complete code documentation (both using Doxygen and updating remote repository) in view of the final 'pencil down'

Blocking issues

  • None in particular

Week 13: 11 August 2014

Work done

  • I have refined GlCoverFlowView class focusing my attention on the two functions 'FirstPos' and 'UpdatePos' in order to define and apply right screen positions to the loaded images
  • In this way, I have finally developed a dynamic CoverFlow visualization mode (see a brief demonstration at ICE-OpenGL CoverFlow): pressing left and right keys loaded images page through along the x direction and, progressively, the selected (centered) image is in evidence respect to the others
  • I have 'scrubbed' the code for the final version and documented all source code with Doxygen special documentation blocks in order to generate relative documentation
  • I have updated the remote repository with both code and README instruction final versions

Future ideas / How can my idea be expanded

First of all the application requires some refinements, as allowing to manage all file formats. Then, a first extension of this project could be its integration in Monteverdi2 software in order to supply also this applicaion to the users. Moreover, taking into account the ESA Sentinel-1 mission, it coulb be useful and interesting expanding these functionalities to data acquired by SAR satellites, in particular to manage large interferometric stacks.