`apply_registration_to_image` task sometimes doesn't copy tables

[jluethi]

I've recently observed an edge case happening to 2 users at the FMI with relatively large multiplexing experiments (200 wells x 2 cycles or 384 wells x 5ish cycles):
When the apply_registration task is run, most images are created correctly. For some reason I don't fully understand yet, in a handful of images it does not copy the ROI tables from the old to the new image though.

This is supposed to happen here:

fractal-tasks-core/fractal_tasks_core/tasks/apply_registration_to_image.py

Line 199 in 0919caf

if table_dict:

It logs every table it copies. For a random handful of images, it doesn't copy any of them though.

If those images are rerun later, they copy the tables correctly though.

What's happening here is that tables are copied from the reference OME-Zarr. But the reference OME-Zarr is also potentially modified. The error happens both when processing the reference OME-Zarr itself, as well as when processing other cycles.

My suspicion is that we have some kind of race condition or file system caching issue here where somehow the task tries to list the tables in the reference cycle and gets an empty list back (=> skips this loop). When we refactor this task to ngio, let's keep an eye on this.

Suspiciously, we've only ever observed this error at FMI, never at UZH (even though this task was in heavy use by APX workflows).

[jluethi]

For users hitting this, the current workaround is to rerun the apply task for just the images that are missing the ROI tables. If overwrite input was chosen, the part of the workflow before also needs to be rerun (often e.g. restarting from the projection task)