Integrating targets and plumber

[sluga]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I have a couple of long-running data pipelines that are triggered when users submit a POST request to a Plumber API.

My current approach uses promises and future.callr to start the pipeline in a background process and immediately return the task ID to the user. So the plumber handler functions include something like this:

task_id <- uuid::UUIDgenerate()
promises::then(
    promises::future_promise(
        {
            # data pipeline:
            step1()
            step2()
            step3()
        }
    ),
    onFulfilled = function(res) {
        record_success(task_id, res)
    },
    onRejected = function(err) {
        record_error(task_id, err)
    }
)
list(task_id = task_id)

How would I implement this if I wanted to use targets?

A minimal adaptation might look like this:

task_id <- uuid::UUIDgenerate()
promises::then(
    promises::future_promise(
        {
            # data pipeline:
             targets::tar_make(
                  script = system.file("my-pipeline.R", package = "my.package"),
                  callr_function = NULL
                  # ... other configuration ...
             )
        }
    ),
    onFulfilled = function(res) {
        record_success(task_id, res)
    },
    onRejected = function(err) {
        record_error(task_id, err)
    }
)
list(task_id = task_id)

Would something else be preferable while preserving the core behavior (return immediately after launching the pipeline)?

[wlandau]

Sounds like a cool pattern.

You might find mirai-based promises appealing.

task_id <- "TASK_123"
promise <- promises::then(
  mirai::mirai(targets::tar_make(callr_function = NULL)),
  onFulfilled = function(result) print(paste(task_id, "success")),
  onRejected = function(error) print(paste(task_id, "error"))
)