External databases for metadata management

[wlandau]

targets uses simple text files for metadata (pipe-separated values). These files can get large in pipelines with many targets (#1390) and appending to them creates overhead in tar_make(). Maybe targets can instead use a DuckDB database for metadata.

[r2evans]

Side note for this: if you have more than the main process writing to the database, you will need to deal with the access-contention. DuckDB supports either multiple readonly readers, or only one writeable connection, you cannot have one connection write while any other connection is reading. See Concurrency.

[wlandau]

Thanks for pointing that out. This may be a deal-breaker for metadata. Redis seems like another good option if the profiling data shows a speedup. Though I don't think I'd make Redis the default because it can be a burden to install.

[r2evans]

I have considered duckdb for many larger-scale projects where having a local quasi-permanent store would be good, but the concurrency issue is a deal-breaker, and not one they seem eager to remedy, unfortunately. Redis would be easy enough for many, but I agree that even as easy as it is, running a redis instance "just for this" might be more than some people prefer.

If you're considering Redis but don't want the server overhead (and perhaps would like on-disk persistence), rrlite can be a good option. Not just because it doesn't require a server, but also because the difference between rrlite and redux is just the initial connection ... otherwise it's intended to be API-compatible with redux, except file-based (or :memory:-based).

I haven't benchmarked it nor verified its concurrency (other than finding seppo0010/rlite#13 (comment)). Admittedly, it hasn't seen commits in many years, I don't know if that means it's awesome-stable or not. richfitx is the author/maintainer (also maintains redux).

[wlandau]

I have been profiling example pipelines, and the bottleneck seems to be reopening the base::file() connection for each line appended to _targets/meta/meta. It would be much faster to keep a persistent open connection, but it may interfere with other connections that read the file. I will test.

[wlandau]

Maintaining a persistent connection seems to reduce execution time from around 60 seconds down to around 23 seconds on an M2 Mac in the following 10000-target pipeline:

library(targets)
tar_option_set(
  controller = crew::crew_controller_local(workers = 25L)
)
list(
  tar_target(datasets, seq_len(1e4), memory = "persistent"),
  tar_target(models, datasets, pattern = map(datasets), retrieval = "main")
)

[wlandau]

There's still a ~30% bottleneck in cat() when appending though.

[wlandau]

Negligible improvement with file(blocking = FALSE).

[wlandau]

There's still a ~30% bottleneck in cat() when appending though.

This is only when targets complete instantaneously.

Moving to a database may be necessary, but on reflection, it is a bit extreme. Converting to a discussion.

[r2evans]

Since you had considered redis, have you looked at rrlite? I don't have your large-scale tests to benchmark, but I'd think that if it performs anywhere near Redis's LPUSH (which is O(1)), it could be good, even if it hasn't seen updates. (Admittedly, I'm mostly just a bringer-of-good-ideas on this one ... I don't use rrlite, and for something like this I would likely use redux with a proper Redis/Valkey instance.)

[wlandau]

Haven't tried rrlite, but it definitely sounds worth looking into.

[lgaborini]

I'm encountering a hard-to-track bug, I'm running a targets pipeline on a Shiny server hosted inside Azure App Service.
The pipeline randomly fails to record its status onto the _targets/meta/meta, possibly due to some race conditions (all perms should be granted, I can also file.create it) or something else going on.

2025-12-25T21:58:05.9091595Z In file.create(to[okay]) :
2025-12-25T21:58:05.9091629Z   cannot create file '/home/_targets/meta/meta', reason 'No such file or directory'

Moving the metadata file to a proper database might be a solution.