Difference between revisions of "Request for Changes-26: MPI Image File Writer"

From OTBWiki
Jump to: navigation, search
([Request for Changes - 24] MPI Image writing)
(New third party modules)
Line 27: Line 27:
 
===== New third party modules =====
 
===== New third party modules =====
 
* MPI (Message Passing Interface)
 
* MPI (Message Passing Interface)
* SPTW (Simple Parallel Riff Writer) developed by David Matthew Mattli at USGC (current maintainer is Kornelijus Survila). Pull request submitted on 15.02.2016 (to make SPTW writing stripped images,only tiled writing was implemented). Currently used is my fork of SPTW.
+
* SPTW (Simple Parallel Tiff Writer) developed by David Matthew Mattli at USGC (current maintainer is Kornelijus Survila). Pull request submitted on 15.02.2016 (to make SPTW writing stripped images,only tiled writing was implemented). Currently used is my fork of SPTW.
  
 
===== New group of modules: MPI =====
 
===== New group of modules: MPI =====

Revision as of 17:05, 14 June 2016

[Request for Changes - 24] MPI Image writing

Status

Summary

Provide a set of parallel image file writers:

  • MPITiffWriter, that use MPI-IO to write one single Geotiff image simultaneously
  • MPIVrtWriter, that writes multiple files simultaneously then produce a VRT

Seamlessly use those writers instead of the standard one when calling an application with mpirun command.

Rationale

Use OTB on HPC architectures

Implementation details

Classes and files

New third party modules
  • MPI (Message Passing Interface)
  • SPTW (Simple Parallel Tiff Writer) developed by David Matthew Mattli at USGC (current maintainer is Kornelijus Survila). Pull request submitted on 15.02.2016 (to make SPTW writing stripped images,only tiled writing was implemented). Currently used is my fork of SPTW.
New group of modules: MPI
A       Modules/MPI/MPIConfig/test/CMakeLists.txt
A       Modules/MPI/MPIConfig/test/otbMPIConfigTestDriver.cxx
A       Modules/MPI/MPIConfig/test/otbMPIConfigTest.cxx
A       Modules/MPI/MPIConfig/include/otbMPIConfig.h
A       Modules/MPI/MPIConfig/otb-module.cmake
A       Modules/MPI/MPIConfig/CMakeLists.txt
A       Modules/MPI/MPIConfig/src/CMakeLists.txt
A       Modules/MPI/MPIConfig/src/otbMPIConfig.cxx
A       Modules/MPI/MPIConfig/README
A       Modules/MPI/MPIVrtWriter/test/CMakeLists.txt
A       Modules/MPI/MPIVrtWriter/test/otbMPIVrtWriterTestDriver.cxx
A       Modules/MPI/MPIVrtWriter/test/otbMPIReadWriteTest.cxx
A       Modules/MPI/MPIVrtWriter/include/otbMPIVrtWriter.h
A       Modules/MPI/MPIVrtWriter/otb-module.cmake
A       Modules/MPI/MPIVrtWriter/CMakeLists.txt
A       Modules/MPI/MPITiffWriter/test/otbMPISPTWReadWriteTest.cxx
A       Modules/MPI/MPITiffWriter/test/CMakeLists.txt
A       Modules/MPI/MPITiffWriter/test/otbMPITiffWriterTestDriver.cxx
A       Modules/MPI/MPITiffWriter/include/otbSimpleParallelTiffWriter.txx
A       Modules/MPI/MPITiffWriter/include/otbSimpleParallelTiffWriter.h
A       Modules/MPI/MPITiffWriter/otb-module.cmake
A       Modules/MPI/MPITiffWriter/CMakeLists.txt
  • MPIConfig module encapsulates basic MPI operations (initialization, errors, etc).
  • MPITiffWriter module writes one GeoTiff file with multiple processes.
  • MPIVrtWriter module writes multiple files, then creates a VRT.
Changes in Wrapper module

Changes are made to seamlessly parallelize all image files writing of existing applications either with MPITiffWriter or MPIVrtWriter, depending on the extension of the output image file. This means that any output image parameter can be written in parallel with MPI, wihtout changing the code of the application.

Tests

Tests added for all three new modules in group MPI.

Tests consists in comparing one image written with the original otb::ImageFileWriter, VS images produced with the MPI image file writer using different tiling scheme.

Documentation

No changes.

Additional notes

Remaining issues are:

  • Licence of SPTW is said "Public Domain" on github. Is this valid for code provenance review and/or debian packaging ?
  • Shall we add OpenMPI to Superbuild ? Probably not.
  • What kind of documentation do we need for this feature ? Section in Software Guide ? Wiki page ?

Not addressed by this RFC:

  • MPI-based persistent filters, for global statistics computation
  • MPI-based segmentation