Request for Comments-18: Factorize CMake configuration

From OTBWiki
Jump to: navigation, search

Status

  • Author: Victor Poughon
  • Submitted on 04.12.2015
  • Open for comments

Content

What changes will be made and why they would make a better Orfeo ToolBox?

Introduction

This RFC concerns otb-devutils only.

Currently, we are using cmake and ctest for nightly integration and testing of multiple projects accross multiples machines. Each build server uses its own configuration and its own builds, with some shared code in otb_common.cmake. With M build configurations and N build servers, our present setup requires N*M cmake configuration files. The main objective of this RFC is to reduce that number to N+M by separating machine specific and build specific configuration.

Problems identified with the current setup:

  • Script maintenance takes a lot of time
  • Adding or editing a build configuration requires editing as many files as there are build servers, making maintenance costly and error prone.
  • Dashboard breaks often
  • Dependency upgrade is troublesome (file paths break, etc.)
  • A lot of configuration is copy-pasted across builds and test sites, and should be factorized whenever possible
Summary of tasks
  • Reorganize otb-devutils/Config into otb-devutils/Config/Sites/ and otb-devutils/Config/Builds/
  • Modify otb_common to be the new ctest entry point (done in proof of concept branch)
  • Migrate all builds
  • Enable more tests by easily composing site and builds (i.e. we can have each server run all available builds, with exceptions)
  • Remove the need for platform specific bash scripts with build lists
Concepts

A build: A configuration specifying which project is compiled and tested (otb, ice, etc.) and all its configuration (compiler, flags, dependencies, Release or Debug, trunk or stable, etc.). Examples of possible builds:

  • OTB-Minimal-gcc
  • OTB-clang-ThirdPartyTrunk
  • OTB-gcc-c++11
  • Monteverdi2-Wextra
  • etc.

A test site: Configuration of a specific server or machine running builds. This configuration can include dashboard root location, dependencies install location, etc. Examples:

  • hulk
  • pc-christophe
  • raoul
  • etc.

A build list: A list of builds indicating which build a test site will run. Ideally, all test sites should run all builds, but this may not be possible for various reasons (dependency not available, windows vs linux, etc). This enables:

  • Selecting which builds to run at a given site by composing site config and build config
  • Making multiple lists. For example, nightly, weekly, continuous, etc.
Configuration files structure

With the modifications proposed by this RFC, otb-devutils/Config would contain: 

   Lists/
           nightly-default.txt
           weekly-default.txt
           nightly-raoul.txt
           nightly-pc-christophe.txt
   Sites/
           default.cmake
           pc-christophe.cmake
           hulk.cmake
           raoul.cmake
           ...
   Builds/
           OTB/
                   stable.cmake
                   trunk.cmake
                   clang.cmake
                   gcc-x.x.cmake
                   c++11.cmake
                   ThirdPartyTrunk.cmake
                   ...
           Ice/
                   stable.cmake
                   clang.cmake
                   trunk.cmake
                   ...
           ITK/
                   stable.cmake
                   trunk.cmake
           OpenJPEG/
           Muparserx/
Standard dashboard layout

This RFC requires a standard dashboard layout common to build servers, located at the configurable CTEST_DASHBOARD_ROOT. 

sources/
        otb
        otb-devutils
        otb-documents
        ice
        monteverdi
        itk
        opencv
        openjpeg
        ...
build/
        otb/
                stable
                trunk
                c++11
                ...
        muparserx/
                stable
                trunk
                ...
        ...
install/
        otb/
                stable
                trunk
                c++11
                ...
        muparserx/
                stable
                trunk
                trunk-c++98
                ...
        ...
logs/
        2015-11-30/
                otb/
                        stable/
                        ...
                itk/
                ...
Technical details

Composing site and build config is achieved by using otb_common.cmake as the entry point to ctest. Two variables containing paths to the site and build configuration files are given to otb_common with ctest -D flag: 

   ctest -S otb_common.cmake -DCTEST_OTB_SITE_CONFIG:PATH=Sites/site.cmake -DCTEST_OTB_BUILD_CONFIG:PATH=Builds/build.cmake

These files define macros (dashboard_hook_init_site and dashboard_hook_init_build) which are then called at the appropriate time in otb_common. Build scripts sizes are minimized by factorizing what can be into otb_common (with possibly defaults), otherwise into site specific configuration (e.g. Sites/raoul.cmake, Sites/pc-christophe.cmake, etc.).

Proof of concept branch

I pushed a proof of concept in otb-devutils branch called factor_config_proof_of_concept.

Using this branch, running the following command will build OTB/nightly.cmake with Sites/pc-christophe.cmake configuration, and submit to dashboard. 

ctest -S otb_common.cmake -V -DCTEST_OTB_SITE_CONFIG:PATH=Sites/pc-christophe.cmake -DCTEST_OTB_BUILD_CONFIG:PATH=Builds/OTB/nightly.cmake

When will those changes be available (target release or date)?

To be determined.

Who will be developing the proposed changes?

Me, but help is welcome.

Community

Comments

List here important comments (possibly links) received.

Support

List here community members that support this RFComments.

Corresponding Requests for Changes

List here links to corresponding Requests for Changes if any.

Retrieved from "https://wiki.orfeo-toolbox.org/index.php?title=Request_for_Comments-18:_Factorize_CMake_configuration&oldid=7169"