Sharing targets through S3

[stelsemeyer]

Hi folks,

First of all: great package! Thanks for creating targets! 😊

Maybe you can help with the following challenge: I have two pipelines, a and b, and I would like to use data of pipeline a within b. Ideally I'd like to use S3 as a backend to share the data as files. I set up the S3 connection as described in the docs, and it works properly. The docs seem to suggest loading the rds files from the other storage as well, but not using some cloud storage as a backend.
I have come up with the following non-working minimal example:

a.R:

library("targets")

tar_option_set(
  resources = tar_resources(
    aws = tar_resources_aws(bucket = "my-bucket-edfihsgerrf")
  )
)

write_data <- function(data) {
  file <- tempfile()
  saveRDS(data, file)
  file
}

list(
  tar_target(output_file, write_data("some-data"), format = "file", repository = "aws")
)

b.R:

library("targets")

tar_option_set(
  resources = tar_resources(
    aws = tar_resources_aws(bucket = "my-bucket-edfihsgerrf")
  )
)

list(
  tar_target(input_file, command = "store-a/objects/output_file", format = "file", repository = "aws")
)

a runs charmingly via tar_make(script = "b.R", store = "store-a").
I can see the output_file in the S3 bucket (as _targets/objects/output_file, no reference to store-a).

Now running tar_make(script = "b.R", store = "store-b") throws an error (coming from tar_assert_path). It seems like I cannot reference the other file:

# • start target input_file
# ✖ error target input_file
# • end pipeline: 0.187 seconds
# Error in tar_throw_run(target$metrics$error) : 
#   _paths_ missing files: store-a/objects/output_file
# Error in `tar_throw_run()`:
# ! callr subprocess failed: _paths_ missing files: store-a/objects/output_file
# Visit https://books.ropensci.org/targets/debugging.html for debugging advice.
# Run `rlang::last_error()` to see where the error occurred.

I have also tried other input file paths, without success.
Given that error, it seems like I am not fully understanding how files and the cloud backends are supposed to work. I saw that you had a discussion about similar issues in the past, as in here.
I would appreciate a hint how to fix this or a suggestions for a workaround. I can also live with a "don't use it like this" though.

Thanks a lot,
Seb

[wlandau]

ropensci-books/targets@807bcde may help a bit (see also https://books.ropensci.org/targets/data.html#local-data-store and #851). When a target is in the cloud (e.g. repository = "aws") it is not in _targets/objects/ and cannot be detected by another format = "file" (even if the downstream target has repository = "aws"). For your case, you could track the upstream object as a URL (with format = "url"). There are other workarounds based on cues.

[mdsumner]

been enjoying and learning tracing through these topics! It seems that the immediate error is because tar_assert_path is only capable of using file.exists? I've had success modifying that to do a check on the s3:// path, but I need a lot more familiarity to explore the implications of that. Is having downstream targets relying on s3 paths something we should aspire to? I wonder if I'm missing more context. I've tried tracking format = "url" but that is rely slow compared to an s3 object check. For now I'm setting a target that is a dataframe of sources+output, and I force this to always run, but shortcut if the s3://output already exists.

[wlandau]

It seems that the immediate error is because tar_assert_path is only capable of using file.exists?

The native S3 integration in targets isn't designed to be split across pipelines in the way OP proposes. tar_assert_path() is one way this shows up, but there are others.

I've tried tracking format = "url" but that is rely slow compared to an s3 object check.

Yes, S3 objects in their own pipeline are queried with a fast LIST API call, but URLs need to be checked individually. To work around that limitation, you might have to build your own CAS system on top of tar_repository_cas().

[mdsumner]

oh! thank you, I had missed that about mixing stores

also, I thought I could return a s3:// address and have that tracked, I guess I'm imagining format = "object" in the S3 sense, but we can only have bare named bytes in objects/ . I can see this all clearly stated now in the book, much appreciate your very fast feedback, super helpful.

I will explore the CAS approach at some point, thanks!

[stelsemeyer]

@wlandau: Thank you! I will give it a try!

Regarding tracking it via url: I would then need to make that bucket public, right? At least it seems to me that if I use the normal https-endpoints, I get 403s.

[wlandau]

Yeah, that's a good point. I am not sure if there is a way to access an object in a private bucket with a URL. If not, then you could always get the hash or last-modified time from a HEAD request and combine that with a tar_cue() or tarchetypes::tar_change() in your pipeline. That HEAD request could use paws, e.g.:

targets/R/utils_aws.R

Lines 28 to 44 in 11c2204

	aws_s3_head <- function(
	key,
	bucket,
	region = NULL,
	endpoint = NULL,
	version = NULL
	) {
	client <- aws_s3_client(endpoint = endpoint, region = region)
	args <- list(
	Key = key,
	Bucket = bucket
	)
	if (!is.null(version)) {
	args$VersionId <- version
	}
	do.call(what = client$head_object, args = args)
	}

[stelsemeyer]

Thanks again for your help! I will take a look at it! 👍