[help] expected behavior of tar_resources_custom_format() with custom tar_format() read method

[brownag]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Thank you for the wonderful targets package.

What is the expected behavior of tar_resources_custom_format() in terms of environment variables visible in the "read" format function?

envvars Named character vector of environment variables. These environment variables are temporarily set just before each call to the storage methods you define in tar_format(). Specific methods like read can retrieve values from these environment variables using Sys.getenv(). Set envvars to NULL to omit entirely.

The documentation and #1263 (comment) suggest that the environment vars are available in read method, but if that were the case the following reprex would return TRUE for both added attributes:

library(targets)

tar_script({
    tar_target(
        a, data.frame(alpha = letters),
        resources = tar_resources(custom = tar_resources_custom_format(c(foo = "bar"))),
        format = tar_format(
            read = function(path) {
                res <- readRDS(path)
                attr(res, "read_has_foo") <- "foo" %in% names(Sys.getenv())
                res
            },
            write = function(object, path) {
                attr(object, "write_has_foo") <- "foo" %in% names(Sys.getenv())
                saveRDS(object, path)
            }
        )
    )
})

tar_make()
#> + a dispatched
#> ✔ a completed [0ms, 198 B]
#> ✔ ended pipeline [77ms, 1 completed, 0 skipped]
tar_load(a)
attributes(a)
#> $names
#> [1] "alpha"
#> 
#> $class
#> [1] "data.frame"
#> 
#> $row.names
#>  [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
#> [26] 26
#> 
#> $write_has_foo
#> [1] TRUE
#> 
#> $read_has_foo
#> [1] FALSE

[wlandau]

That's a tricky one. Those envvars are available during tar_make() for both read and write, but not necessarily for tar_read() outside a pipeline. Since environment variables could contain secrets like security credentials, targets does not save them. So you'll have to manually set the resources in the interactive R session where you want to run tar_read().

tar_option_set(resources = tar_resources(
  custom = tar_resources_custom_format(c(foo = "bar"))
))
object <- tar_read(a)
attr(object, "write_has_foo")
#> [1] TRUE
attr(object, "read_has_foo")
#> [1] TRUE

[brownag]

Thanks so much @wlandau , that explains the "why" of the behavior, which makes perfect sense. I wasn't fully considering that the read method was being called in two different contexts, though I figured the difference was due to concerns about leaking secrets or similar.

This means tar_resources_custom_format() won't work for what I need, so I will go back to constructing a closure with the values I want baked in, (as you suggested in #1263 (comment)) something like:

library(targets)

tar_script({
    .create_format <- function(foo_val) {
        tar_format(
            read = eval(substitute(function(path) {
                res <- readRDS(path)
                attr(res, "read_foo_val") <- foo_val
                res
            })),
            write = eval(substitute(function(object, path) {
                attr(object, "write_foo_val") <- foo_val
                saveRDS(object, path)
            }))
        )
    }
    
    tar_target(
        a, data.frame(alpha = letters),
        format = .create_format(foo_val = "bar")
    )
})

tar_make()
#> + a dispatched
#> ✔ a completed [1ms, 198 B]
#> ✔ ended pipeline [72ms, 1 completed, 0 skipped]
tar_load(a)
attributes(a)
#> $names
#> [1] "alpha"
#> 
#> $class
#> [1] "data.frame"
#> 
#> $row.names
#>  [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
#> [26] 26
#> 
#> $write_foo_val
#> [1] "bar"
#> 
#> $read_foo_val
#> [1] "bar"

For my use case the above approach is probably more flexible anyway, as the user will not need to customize these values much, or if they do they could be handled by an argument in the target factory for the custom format, which is, I think, preferable to having them set custom resources for interactive use.

[wlandau]

Unfortunately, closures get dropped because these functions get saved as text. I'd recommend having a look at the substitute argument of tar_format().