Writing large images

For a long time, we experienced slow-down when writing very large images such as pan-sharpened VHR products in geoTiff files (40 000 x 40 000 pixels, 4 bands, 16bits). With the new control on how tiff file is generated provided by ExtendedFileName, we tried to understand where the bottleneck was.

The Gdal cache overflow issue

Using the extended filenames options (gdal:co:TILED, gdal:co:BLOCKXSIZE, gdal:co:BOCKYSIZE) following piece of code:

  for(unsigned int lenght = lstart; lenght < lstop; lenght+=lstep)
    {
    ImageType::Pointer image = ImageType::New();
    ImageType::RegionType region;
    ImageType::SizeType size;
    size[0] = lenght;
    size[1] = height;
    
    region.SetSize(size);

    image->SetNumberOfComponentsPerPixel(nbBands);
    image->SetRegions(region);
    image->Allocate();
    image->FillBuffer(black);

    itk::TimeProbe chrono;

    WriterType::Pointer writer = WriterType::New();
    writer->SetInput(image);
    writer->SetFileName(outfname);
    
    if(mode == "tilestream")
      {
      writer->SetAutomaticTiledStreaming();
      }
    else if(mode =="stripstream")
      {
      writer->SetAutomaticStrippedStreaming();
      }
    else
      {
      std::cerr<<"Unknown mode "<<mode<<", available are tilestream and stripstream"<<std::endl;
      return EXIT_FAILURE;
      }


    chrono.Start();
    writer->Update();
    chrono.Stop();

    ofs<<lenght<<"\t"<<chrono.GetMeanTime()<<std::endl;
    }

We tried to profile the writing time with respect to the width of the image, for each streaming mode and tiff layout mode. The following graph shows the results of the profiling :

As shown in this figure, it appears that all combination of streaming mode and tiff file mode lead to the same sudden drop in performances at the same point. After further investigations, it appears that this point correspond to the unitary block becoming larger than the allowed gdal cache (default is 40 Mb in 1.8/1.9 releases). If the cache is extended to 256 Mb, the problem disappears (red curve). So if you are planning to write large image file with OTB, it is a good idea to extend gdal max allowed cache by using the GDAL_CACHEMAX environment variable:

export GDAL_CACHEMAX=256

When experiencing strange performance drop like this one, a good thing to do is to run the program with the environment variable

CPL_DEBUG=ON

If the program outputs message like :

GDAL: Potential thrashing on band 1 of test.tif. 

this usually means you should increase your cache size by setting GDAL_CACHEMAX.

Never do tiled streaming and write stripped GeoTiff

In this section, we will compute the maximum expected Gdal cache size (in pixels) with respect to the size of the streaming buffer (in pixels), for each of GeoTiff mode (tiled and stripped) and for each of the streaming mode (tiled and stripped). Please note that this formulas are only very simple estimation and does not account for optimization or more clever behaviour implemented in Gdal. As such, it can be seen as an upper bound of the expected Gdal cache size.

Let w denote the image width, streamsize the size of the stream buffer in pixels, tilesize the size of the GeoTiff tiles in tiled mode, stripsize the height of the GeoTiff strips in stripped mode.

Cache size of stripped streaming when writing to stripped GeoTiff

cachesize = ceil(ceil(streamsize/width)/stripsize) * width * stripsize

Complexity is

O(streamsize)

Cache size of stripped streaming when writing to tiled GeoTiff

cachesize = ceil(ceil(streamsize/width)/tilesize) * ceil(width/tilesize) * tilesize * tilesize

Complexity is

O(streamsize)

Cache size of tiled streaming when writing to stripped GeoTiff

cachesize = ceil(sqrt(streamsize)/stripsize) * stripsize * width

Complexity is

O(sqrt(streamsize)*width)

Cache size of tiled streaming when writing to tiled GeoTiff

ceil(sqrt(streamsize)/tilesize) * ceil(sqrt(streamsize)/tilesize) * tilesize * tilesize

Complexity is

O(streamsize)

So, long story short, the tiled to strip case is the only one with a linear complexity with respect to the width of the image.

Increasing available RAM can make it worse

Using the above formulas, we can simulate the expected maximum Gdal cache with respect to the size of stream. For a 50 000 pixels wide image, with 4 bands and 16 bits precision, with 1 lines strip (for stripped GeoTiff) and 256x256 tiles (for tiled GeoTiff), we obtain the following curves:

This enforces the conclusion of the former section : never write stripped GeoTiff while streaming with tiles. But one can also notice that the other curves are quite steep : with a stream buffer of 1000 Mb, the expected Gdal maximum cache is above 100 Mb for all modes. Since the maximum allowed GDAL_CACHEMAX value is 2047, and this only accounts for writing operation, it is probably a good idea to keep the RAM parameter value in a reasonable range.

Please note that there are no direct connexion between the size of the stream in Mb and the RAM parameter exposed by most application, since the latter is used to compute the former by estimating the footprint of the whole OTB pipeline in use.