[help] About accelerating geotargets loading large raster ?

[reyman]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi,

I have some difficulties to found the good way to load and use large raster with targets, geotargets and crew.

Using this simple plan, and a "not so big" 3GB raster, i takes looooooot of time for load_height target to complete.
The raster is available here :
https://ent.normandie-univ.fr/filex/get?k=N6OrrcT3OB54wxz3Xys

tar_plan(
   tar_target(
      height_file,
      file.path("./data_folder/auxiliary/height_nouvelle_aquitaine.tif"),
      format = "file",
      ),
    
    tar_terra_rast(load_height,
                   terra::rast(height_file),
                   repository = "local",
                  ),
)

Is there some way to reduce the process of rast()

Any help appreciated :)
Src

[wlandau]

From profiling, it looks like the slowness comes from terra::writeRaster(). Performance improvements would have to come from the terra package itself (or possibly from geotargets if there are more optimal ways to configure the function call to terra::writeRaster()). Nothing targets can do, I'm afraid.

[brownag]

Hi, just to add to @wlandau's assessment here.

The philosophy of geotargets results in essentially copying your file sources to the target store... this means that you are re-writing your external file "./data_folder/auxiliary/height_nouvelle_aquitaine.tif" into "_targets/objects" subfolder.

This does result in some write overhead, but we think it improves reproducibility in the general case

In your case the overhead is perhaps unneeded because you aren't processing the source file in any way--but in the event you were doing some calculations on it you would want to be able to store those results separately in the target store.

To Will's point, there are some options you can tinker with in terms of arguments passed to terra::writeRaster() via tar_terra_rast():

• gdal argument, "COMPRESS" creation option: Sometimes the default "LZW" compression used by terra can be a bottleneck. LZW is generally a reasonable choice in terms of balance of performance and file size, but depending on your data you may choose to use a different creation option for a different compression algorithm.

  • These days, with large and very performant SSDs, some folks find it is a little faster to write files with no compression when storage is not a limiting factor. Of course with no compression the file will generally be much larger. You can specify different compression arguments via the gdal argument to tar_terra_rast() which is passed to writeRaster(). E.g. gdal="COMPRESS=NONE". You can see all the choices GDAL offers for the default GeoTIFF driver used by tar_terra_rast() here: https://gdal.org/en/stable/drivers/raster/gtiff.html#creation-options

• datatype argument: If you are willing to accept some loss of precision, or the values in your raster could be losslessly encoded with a lower bit depth, you could try altering the datatype argument. For instance, many folks don't realize data that could be just as well represented with a byte or 16-bit integer (signed or unsigned) are actually stored in their files as 32-bit floating point, which is much larger. Optimizing the datatype will reduce the file size and write time, and also can make compression more efficient.

Alternately, if you are willing to have your pipeline depend on the external file, you could consider using tar_terra_vrt() instead of tar_terra_rast(). This creates a GDAL Virtual Raster Dataset (VRT) from your source file--and thus runs essentially instantaneously as the target object is just a SpatRaster pointing to XML file that references your original file path. You can do this as:

    tar_terra_vrt(load_height,
                   terra::rast(height_file),
                   repository = "local",
                  )

We originally intended tar_terra_vrt() for use with tiled datasets in the target store such as those created by tar_terra_tiles(), but this should work well for your case. Note that you should always use the format="file" target (as you have in example) so that any changes external to the target store will invalidate the downstream targets.

If you run into any issues with any of the above, please let us know over on geotargets issues: https://github.com/ropensci/geotargets/issues

[reyman]

Thanks a lot @wlandau @brownag for your quick answer and the solution proposed.

I'm a little lost with the different option to manage data sources when theyr are needed during dynamic branching, over crew workers.

To better illustrate the workflow :

[brownag]

If you have not yet seen it, you may find this vignette "Dynamic branching with raster tiles" helpful as a starting point. https://ropensci.github.io/geotargets/articles/tar_terra_tiles.html; and of course the targets user guide for dynamic branching: https://books.ropensci.org/targets/dynamic.html

From your schematic it looks like you need to be branching over over years and tiles--something like that can be handled relatively easily with existing tools.

If you have further specific questions/errors/unexpected behaviors with the geotargets specific parts we could help you out over on the geotargets repo. @Aariq has more experience with crew in context of geotargets than I do

[reyman]

Yep, i have already some part of the worflow actually, iterating over tile are not a problem because they are small (250 mb each), this is more the link with load_height and load_slope into the dynamic branching that break/crash my R Kernel actually ... I will try with VRT :)

Edit : Yup, really more faster with VRT ! But what's the behavior with multiple workers on the same machine ?

load_raster <- function(tiles) {
    # Load data
    classif <- tiles[[1]][1]
    conf <- tiles[[1]][2]
    classif <- terra::rast(classif[[1]])
    conf <- terra::rast(conf[[1]])
    return(sprc(classif,conf))
    }

tar_plan(

   # A 9 gb file used during workflow pipeline applied to each tile
    tar_target(
      height_file,
      file.path("./data_folder/auxiliary/height_nouvelle_aquitaine.tif"),
      format = "file",
      ),

   # A 3 gb file used during workflow pipeline applied to each tile      
    tar_target(
      slope_file,
      file.path("./data_folder/auxiliary/slope_nouvelle_aquitaine.tif"),
      format = "file",
      ),
    
    tar_terra_rast(load_height,
                   terra::rast(height_file),
                   memory="persistent",
                   repository = "local",
                  ),
    
    tar_terra_rast(load_slope,
                   terra::rast(slope_file),
                   memory="persistent",
                   repository = "local",
                  ),
    
   # iterate on years folder
   tar_target(list_years,
          find_years("./data_folder")),
    
   # iterate on tiles stored in each year folder
    tar_target(list_tiles,
               read_raster(list_years),
               pattern=map(list_years)),

   # load as sprc the two rasters associated to each tile ( ie tile1_classif.tiff, tile1_conf.tiff)
    tar_terra_sprc(tile_pipeline,
               load_raster(list_tiles),
              pattern=map(list_tiles)
              ),
  
   # running steps described into _process_tile_pipeline_ function with given data (ie => _tile1_classif, tile1_conf, load_slope, load_height_ for example)     
    tar_terra_sprc(tiles,
        process_tile_pipeline(
        ...
        slope_img=load_slope,
        height_img=load_height,
        ...
        ),
        pattern=map(tile_pipeline))
 )