[help] tar_make() hangs "querying #### local files in .targets_stores/.../objects"

[dblodgett-usgs]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Long time drake / targets user here. I run workflows with hundreds of branches that each have 10s of targets. I'm struggling to identify the source of a performance issue that is really holding my iteration back.

After running tar_make() I get the message querying #### local files in .targets_stores/.../objects and the pipeline just hangs -- maybe recalculating file hashes? When the message is displayed my computer is marginally locked up and I cannot stop the R process in the normal way. I have to force quit the app.

Once I force quit, y computer becomes responsive again.

Has anyone encountered this? Any idea what I can do to fix my targets store so it doesn't occur?

[wlandau]

The "querying n local files" step happens in targets:::runtiem_set_file_info(). The slow step is a (wrapped) file.info() call:

targets/R/class_runtime.R

Line 149 in 4d878f7

info <- file_info(

For most large pipelines, querying all the files together can actually speed up the pipeline. If you query them one by one, you'll get a lot more operating system overhead.

If that step is slow, you might try tar_prune() to remove unnecessary files from _targets/objects/. The only caveats with that are (1) if a collaborator is running a pipeline on the same data store as you, or (2) if you are using content-addressable storage (tar_repository_cas() e.g.).

Slowness might also come from low disk speed, e.g. if the data store lives on a Windows network drive instead of your local machine.

[dblodgett-usgs]

Thank you for the follow up! I noticed that something was thrashing my hard drive -- some kind of ghost process as it wasn't showing up in the windows Task Manager. So I think this was actually caused by very high disc latency / blocking.

I'll mark this as answered. Hopefully this question helps someone else who runs into something similar. Again, thank you for your time responding!!