Difference between revisions of "Request for Changes-26: MPI Image File Writer"

From OTBWiki
Jump to: navigation, search
(MPI Image file writer)
(Documentation)
 
(27 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== [Request for Changes - 24] MPI Image file writer ==
+
== [Request for Changes - 24] MPI Image writing ==
  
 
=== Status ===  
 
=== Status ===  
  
 
* Author: Rémi Cresson
 
* Author: Rémi Cresson
* Additional Contributors: David Matthew Mattli
+
* Additional contributors: Julien Michel, Emmanuelle Sarazin
 
* Submitted on 11.02.2016
 
* Submitted on 11.02.2016
* Proposed target release 5.4
+
* Proposed target release 5.6
 +
* Branch: https://git.orfeo-toolbox.org/otb.git/shortlog/refs/heads/rfc-26-mpi_writer
  
 
=== Summary ===
 
=== Summary ===
  
Provide a parallel image file writer, which use MPI-IO to write one image with multiple processing nodes simultaneously
+
Provide a set of parallel image file writers:
 +
* MPITiffWriter, that use MPI-IO to write one single Geotiff image simultaneously
 +
* MPIVrtWriter, that writes multiple files simultaneously then produce a VRT
 +
 
 +
Seamlessly use those writers instead of the standard one when calling an application with mpirun command.
  
 
=== Rationale ===
 
=== Rationale ===
Line 20: Line 25:
 
==== Classes and files ====
 
==== Classes and files ====
  
otbMPIImageFileWriter.h (or something like this?)
+
===== New third party modules =====
otbMPIImageFileWriter.hxx
+
* MPI (Message Passing Interface)
otbMPIHelpers.h (containing some bridge definitions between OTB and MPI, like data types.)
+
* SPTW (Simple Parallel Tiff Writer) developed by David Matthew Mattli at USGC (current maintainer is Kornelijus Survila). Pull request submitted on 15.02.2016 (to make SPTW writing stripped images,only tiled writing was implemented). Currently used is my fork of SPTW.
  
The actual writer writes only GeoTiffs. It relies on the SPTW (Simple Parallel Tiff Writer) [https://github.com/dmm/prasterblaster] developed by David Matthew Mattli.
+
===== New group of modules: MPI =====
I have modified the SPTW to make it writing stripped images (only tiled writing was implemented).
+
<pre>
Maybe it could be convenient to refactor SPTW into the writer class and add David to the contributor list.
+
A      Modules/MPI/MPIConfig/test/CMakeLists.txt
 +
A      Modules/MPI/MPIConfig/test/otbMPIConfigTestDriver.cxx
 +
A      Modules/MPI/MPIConfig/test/otbMPIConfigTest.cxx
 +
A      Modules/MPI/MPIConfig/include/otbMPIConfig.h
 +
A      Modules/MPI/MPIConfig/otb-module.cmake
 +
A      Modules/MPI/MPIConfig/CMakeLists.txt
 +
A      Modules/MPI/MPIConfig/src/CMakeLists.txt
 +
A      Modules/MPI/MPIConfig/src/otbMPIConfig.cxx
 +
A      Modules/MPI/MPIConfig/README
 +
A      Modules/MPI/MPIVrtWriter/test/CMakeLists.txt
 +
A      Modules/MPI/MPIVrtWriter/test/otbMPIVrtWriterTestDriver.cxx
 +
A      Modules/MPI/MPIVrtWriter/test/otbMPIReadWriteTest.cxx
 +
A      Modules/MPI/MPIVrtWriter/include/otbMPIVrtWriter.h
 +
A      Modules/MPI/MPIVrtWriter/otb-module.cmake
 +
A      Modules/MPI/MPIVrtWriter/CMakeLists.txt
 +
A      Modules/MPI/MPITiffWriter/test/otbMPISPTWReadWriteTest.cxx
 +
A      Modules/MPI/MPITiffWriter/test/CMakeLists.txt
 +
A      Modules/MPI/MPITiffWriter/test/otbMPITiffWriterTestDriver.cxx
 +
A      Modules/MPI/MPITiffWriter/include/otbSimpleParallelTiffWriter.txx
 +
A      Modules/MPI/MPITiffWriter/include/otbSimpleParallelTiffWriter.h
 +
A      Modules/MPI/MPITiffWriter/otb-module.cmake
 +
A      Modules/MPI/MPITiffWriter/CMakeLists.txt
 +
</pre>
  
==== Applications ====
+
* MPIConfig module encapsulates basic MPI operations (initialization, errors, etc).
 +
* MPITiffWriter module writes one GeoTiff file with multiple processes.
 +
* MPIVrtWriter module writes multiple files, then creates a VRT.
  
No changes.
+
===== Changes in Wrapper module =====
 +
 
 +
Changes are made to seamlessly parallelize all image files writing of existing applications either with MPITiffWriter or MPIVrtWriter, depending on the extension of the output image file. This means that any output image parameter can be written in parallel with MPI, wihtout changing the code of the application.
  
 
==== Tests ====
 
==== Tests ====
  
Tests could consist in comparing one image writed with the original otb::ImageFileWriter to images produced with the parallel image file writer using different tiling scheme, strategies, etc.
+
Tests added for all three new modules in group MPI.
 +
 
 +
Tests consists in comparing one image written with the original otb::ImageFileWriter, VS images produced with the MPI image file writer using different tiling scheme.
 +
 
 +
Git branch is added to tested feature branches. Among dashboard platforms, pc-christophe and hulk have the MPI libraries installed and MPI modules enabled, while the others do not.
  
 
==== Documentation ====
 
==== Documentation ====
  
No changes.
+
* What kind of documentation do we need for this feature ? Section in Software Guide ? Wiki page ? A short section in Cookbook could be useful.
  
 
=== Additional notes ===
 
=== Additional notes ===
  
List remaining open issues if any, and additional notes.
+
Remaining issues are:
 +
 
 +
* Copyright of MPIVrtWriter should be IRSTEA (Remi Cresson). Copyright notice should be updated accordingly.
 +
* Licence of SPTW is said "Public Domain" on github. Is this valid for code provenance review and/or debian packaging ?
 +
* Both MPIVrtWriter and MPITiffWriter include gdal headers in their headers. Which means that binary packages built with MPI ON will depend on gdal headers. Probably not very important since MPI is mainly intended for cluster arch, where people will rebuild OTB themselves anyway.
 +
* Shall we add OpenMPI to Superbuild ? Probably not.
 +
 
 +
Not addressed by this RFC:
 +
* MPI-based persistent filters, for global statistics computation
 +
* MPI-based segmentation

Latest revision as of 14:02, 1 July 2016

[Request for Changes - 24] MPI Image writing

Status

Summary

Provide a set of parallel image file writers:

  • MPITiffWriter, that use MPI-IO to write one single Geotiff image simultaneously
  • MPIVrtWriter, that writes multiple files simultaneously then produce a VRT

Seamlessly use those writers instead of the standard one when calling an application with mpirun command.

Rationale

Use OTB on HPC architectures

Implementation details

Classes and files

New third party modules
  • MPI (Message Passing Interface)
  • SPTW (Simple Parallel Tiff Writer) developed by David Matthew Mattli at USGC (current maintainer is Kornelijus Survila). Pull request submitted on 15.02.2016 (to make SPTW writing stripped images,only tiled writing was implemented). Currently used is my fork of SPTW.
New group of modules: MPI
A       Modules/MPI/MPIConfig/test/CMakeLists.txt
A       Modules/MPI/MPIConfig/test/otbMPIConfigTestDriver.cxx
A       Modules/MPI/MPIConfig/test/otbMPIConfigTest.cxx
A       Modules/MPI/MPIConfig/include/otbMPIConfig.h
A       Modules/MPI/MPIConfig/otb-module.cmake
A       Modules/MPI/MPIConfig/CMakeLists.txt
A       Modules/MPI/MPIConfig/src/CMakeLists.txt
A       Modules/MPI/MPIConfig/src/otbMPIConfig.cxx
A       Modules/MPI/MPIConfig/README
A       Modules/MPI/MPIVrtWriter/test/CMakeLists.txt
A       Modules/MPI/MPIVrtWriter/test/otbMPIVrtWriterTestDriver.cxx
A       Modules/MPI/MPIVrtWriter/test/otbMPIReadWriteTest.cxx
A       Modules/MPI/MPIVrtWriter/include/otbMPIVrtWriter.h
A       Modules/MPI/MPIVrtWriter/otb-module.cmake
A       Modules/MPI/MPIVrtWriter/CMakeLists.txt
A       Modules/MPI/MPITiffWriter/test/otbMPISPTWReadWriteTest.cxx
A       Modules/MPI/MPITiffWriter/test/CMakeLists.txt
A       Modules/MPI/MPITiffWriter/test/otbMPITiffWriterTestDriver.cxx
A       Modules/MPI/MPITiffWriter/include/otbSimpleParallelTiffWriter.txx
A       Modules/MPI/MPITiffWriter/include/otbSimpleParallelTiffWriter.h
A       Modules/MPI/MPITiffWriter/otb-module.cmake
A       Modules/MPI/MPITiffWriter/CMakeLists.txt
  • MPIConfig module encapsulates basic MPI operations (initialization, errors, etc).
  • MPITiffWriter module writes one GeoTiff file with multiple processes.
  • MPIVrtWriter module writes multiple files, then creates a VRT.
Changes in Wrapper module

Changes are made to seamlessly parallelize all image files writing of existing applications either with MPITiffWriter or MPIVrtWriter, depending on the extension of the output image file. This means that any output image parameter can be written in parallel with MPI, wihtout changing the code of the application.

Tests

Tests added for all three new modules in group MPI.

Tests consists in comparing one image written with the original otb::ImageFileWriter, VS images produced with the MPI image file writer using different tiling scheme.

Git branch is added to tested feature branches. Among dashboard platforms, pc-christophe and hulk have the MPI libraries installed and MPI modules enabled, while the others do not.

Documentation

  • What kind of documentation do we need for this feature ? Section in Software Guide ? Wiki page ? A short section in Cookbook could be useful.

Additional notes

Remaining issues are:

  • Copyright of MPIVrtWriter should be IRSTEA (Remi Cresson). Copyright notice should be updated accordingly.
  • Licence of SPTW is said "Public Domain" on github. Is this valid for code provenance review and/or debian packaging ?
  • Both MPIVrtWriter and MPITiffWriter include gdal headers in their headers. Which means that binary packages built with MPI ON will depend on gdal headers. Probably not very important since MPI is mainly intended for cluster arch, where people will rebuild OTB themselves anyway.
  • Shall we add OpenMPI to Superbuild ? Probably not.

Not addressed by this RFC:

  • MPI-based persistent filters, for global statistics computation
  • MPI-based segmentation