Best practices for Shiny apps that contain targets pipelines

[wlandau]

I think there are two challenges:

1. Writing the _targets.R file. I think the app developer can write a mostly completed _targets.R template with glue patterns to plug in various user inputs. (Is this easy enough for most people?)
2. Persistent storage. I just learned about tools::R_user_dir(), which looks perfect for storing the app user's _targets.R file and _targets/ data for multiple sessions.
3. Working directories. Pretty much everything targets-related requires withr::with_dir() to run in the user's session directory.

Thinking about writing a new shinytargets package to automate a bunch of this. Looping in @rpodcast.

[wlandau]

Also cc @richardbayes

[wlandau]

On second thought, these issues can be addressed in targets itself. Tidy evaluation in tar_script() could take care of (1). For (2) and (3), maybe we need to relax targets' working directory paternalism and allow the user to set the directory where _targets.R and _targets/ are located.

[gorkang]

+1 to "relax targets' working directory paternalism". I've encountered cases where, in the same project, I rather have a set of different _targets.R files, with different and generally unrelated processes, than a single more complex big pipeline.

I can imagine how having a targets folder with _targets_process1.R and _targets_process2.R, and the ability to call any of them through a slightly more flexible tar_make() would make those cases easier to manage.

[wlandau]

In your case, you could maintain a central _targets.R that sources other scripts to define the pipeline.

# _targets.R
library(targets)
source("R/functions.R")
source("R/pipeline_option_1.R")
pipeline_option_1 # list of targets defined in R/pipeline_option_1.R

For Shiny compatibility, I think we can build what we need with parameterized _targets.R files. Details at #291.

[gorkang]

Thanks for your response @wlandau

Maybe I am not fully understanding your suggestion. I want to be able to run only one of two distinct although overlapped lists of targets. Just to define a little more this user case. I have two sources of data:

1. Potential participants fill a form to register for the study. They answer demographic questions and a socioeconomic survey
2. A subset of (1) are invited to complete the full study.

I want to be able to easily do two different things (one or the other, and generally not both at the same time):

A. Generate a report combining the information in (1) with some steps of the processing done with (2)
B. Run the full data preparation, reports on the sanity of the data, etc.

The end result for (A) is a html report with information about the progress of the data collection, how many candidates can invite to fill the missing slots, and the information to pay participants that completed (2). I need to combine all the information collected in (1) and a small subset of the data collected in (2). This is a relatively fast process.

The end result for (B) is a clean data frame with all the tasks from (2) and (1). A longer and more time-consuming process.

Right now I have (A) integrated inside the "big" (B) pipeline, but I'd love to have the option to run tar_make("targets/_target_A.R") or tar_make("targets/_target_B.R") depending on the needs.

[wlandau]

It sounds like targets from (1) are connected to targets from (2), so I recommend you keep both in the same pipeline. The names argument of tar_make() allows you to select subsets of targets to run (plus their upstream dependencies).

In situations where you have two different mutually exclusive or totally independent lists of targets, you could create the target lists themselves in two different scripts.

# file R/pipeline1.R
pipeline <- list(
  tar_target(data1, run_data1()),
  tar_target(model1, run_model1(data1))
)

# file R/pipeline2.R
pipeline <- list(
  tar_target(data2, run_data2()),
  tar_target(model2, run_model2(data1))
)

And then modify _targets.R to switch between the two as needed. Here is the _targets.R file for pipeline 1:

# file _targets.R for pipeline 1
library(targets)
source("R/pipeline1.R)
pipeline

For pipeline 2, change the script you source to get the pipeline object.

# file _targets.R for pipeline 2
library(targets)
source("R/pipeline2.R) # Change this line to switch to a different pipeline
pipeline

[wlandau]

The issues here are reproducibility and technical debt. If you look at a _targets.R file, you know the project starts from there, and you know it is in the working directory where you have to run tar_make(). If we allow customization, it will be misappropriated and create unforeseen reproducibility and technical debt issues.