Merge pull request #876 from fractal-analytics-platform/874_updated_tasks_info

Add tasks docs_info

Author: tcompa

.github/workflows/ci_poetry.yml

@@ -87,8 +87,17 @@ jobs:
       - name: Install dependencies (including fractal-tasks extra)
         run: poetry install --with dev --without docs --no-interaction -E fractal-tasks
 
-      - name: Check manifest task metadata
-        run: poetry run python fractal_tasks_core/dev/check_manifest.py
+      - name: Check if manifest has changed
+        run: |
+          poetry run python fractal_tasks_core/dev/create_manifest.py
+          echo "*.json diff=json" >> .gitattributes && git config diff.json.textconv "jq --sort-keys '.' \$1"
+          git diff ./fractal_tasks_core/__FRACTAL_MANIFEST__.json
+          if [ -n "$(git diff --exit-code ./fractal_tasks_core/__FRACTAL_MANIFEST__.json)" ]; then
+          echo "__FRACTAL_MANIFEST__.json has changed. Please run 'poetry run python fractal_tasks_core/dev/create_manifest.py' and commit the changes."
+            exit 1
+          else
+            echo "__FRACTAL_MANIFEST__.json has not changed."
+          fi
 
       - name: Cache Pooch folder
         id: cache-pooch-folder

CHANGELOG.md

@@ -1,5 +1,11 @@
 **Note**: Numbers like (\#123) point to closed Pull Requests on the fractal-tasks-core repository.
 
+# 1.3.4
+
+* Manifest creation:
+    * Support providing `docs_info=file:task_info/description.md` (\#876).
+    * Deprecate `check_manifest.py` module, in favor of additional GitHub action steps (\#876).
+
 # 1.3.3
 
 * Add new metadata (authors, category, modality, tags) to manifest models and to tasks (\#855).

fractal_tasks_core/__FRACTAL_MANIFEST__.json

@@ -22,6 +22,7 @@
     create_schema_for_single_task,
 )
 from fractal_tasks_core.dev.lib_task_docs import create_docs_info
+from fractal_tasks_core.dev.lib_task_docs import read_docs_info_from_file
 
 
 logging.basicConfig(level=logging.INFO)
@@ -126,11 +127,19 @@ def create_manifest(
                     task_dict[f"args_schema_{kind}"] = schema
 
         # Update docs_info, based on task-function description
-        docs_info = create_docs_info(
-            executable_non_parallel=task_obj.executable_non_parallel,
-            executable_parallel=task_obj.executable_parallel,
-            package=package,
-        )
+        docs_info = task_dict.get("docs_info")
+        if docs_info is None:
+            docs_info = create_docs_info(
+                executable_non_parallel=task_obj.executable_non_parallel,
+                executable_parallel=task_obj.executable_parallel,
+                package=package,
+            )
+        elif docs_info.startswith("file:"):
+            docs_info = read_docs_info_from_file(
+                docs_info=docs_info,
+                task_list_path=task_list_module.__file__,
+            )
+
         if docs_info is not None:
             task_dict["docs_info"] = docs_info
         if docs_link is not None:

fractal_tasks_core/dev/check_manifest.py

@@ -57,7 +57,7 @@ def create_docs_info(
     executable_non_parallel: Optional[str] = None,
     executable_parallel: Optional[str] = None,
     package: str = "fractal_tasks_core",
-) -> list[str]:
+) -> str:
     """
     Return task description based on function docstring.
     """
@@ -81,3 +81,38 @@ def create_docs_info(
     docs_info = "".join(docs_info)
     logging.info("[create_docs_info] END")
     return docs_info
+
+
+def read_docs_info_from_file(
+    *,
+    docs_info: str,
+    task_list_path: str,
+) -> str:
+    """
+    Return task description based on the content of a file.
+
+    An example of valid argument is
+    ```
+    docs_info = "file:relative/path/info.md"
+    ```
+    where the path is relative to the folder where `task_list.py` is.
+    """
+    logging.info("[read_docs_info_from_file] START")
+
+    # Preliminary checks
+    if not docs_info.startswith("file:"):
+        raise ValueError(f"Invalid docs_info='{docs_info}'.")
+    relative_path = Path(docs_info[5:])
+    if relative_path.is_absolute():
+        raise ValueError(
+            f"Invalid docs_info='{docs_info}' (path must be relative)."
+        )
+
+    base_path = Path(task_list_path).parent
+    docs_path = (base_path / relative_path).as_posix()
+    logging.info(f"[read_docs_info_from_file] Reading docs from {docs_path}")
+    with open(docs_path, "r") as f:
+        docs_info = f.read()
+    logging.info("[read_docs_info_from_file] END")
+
+    return docs_info

fractal_tasks_core/dev/create_manifest.py

@@ -0,0 +1,8 @@
+### Purpose
+- **Applies pre-calculated registration** transformations to images in an **HCS** OME-Zarr dataset, aligning all acquisitions to a specified reference acquisition.
+- **Masks regions not included** in the registered ROI table and aligns both intensity and label images.
+- Replaces the non-aligned image with the newly aligned image in the dataset if `overwrite input` is selected.
+- Typically used as the third task in a workflow, following `Calculate Registration (image-based)` and `Find Registration Consensus`.
+
+### Limitations
+- If `overwrite input` is selected, the non-aligned image is permanently deleted, which may impact workflows requiring access to the original images.

fractal_tasks_core/dev/lib_task_docs.py

@@ -0,0 +1,9 @@
+### Purpose
+- **Computes image-based registration** transformations for acquisitions in **HCS** OME-Zarr datasets.
+- Processes images grouped by well, under the assumption that each well contains one image per acquisition.
+- Calculates transformations for **specified regions of interest (ROIs)** and stores the results in the corresponding ROI table.
+- Typically used as the first task in a workflow, followed by `Find Registration Consensus` and optionally `Apply Registration to Image`.
+
+### Limitations
+- Supports only HCS OME-Zarr datasets, leveraging their acquisition metadata and well-based image grouping.
+- Assumes each well contains a single image per acquisition.

fractal_tasks_core/dev/task_info/apply_registration_to_image.md

@@ -0,0 +1,10 @@
+### Purpose
+- **Segments images using Cellpose models**.
+- Supports both **built-in Cellpose models** (shipped with Cellpose) and **user-trained models**.
+- Accepts dual image input for segmentation.
+- Can process **arbitrary regions of interest (ROIs)**, including whole images, fields of view (FOVs), or masked outputs from prior segmentations, based on corresponding ROI tables.
+- Provides access to all advanced Cellpose parameters.
+- Allows custom rescaling options per channel, particularly useful for sparse images.
+
+### Limitations
+- Compatible only with Cellpose 2.x models; does not yet support 3.x models.

fractal_tasks_core/dev/task_info/calculate_registration_image_based.md

@@ -0,0 +1,8 @@
+### Purpose
+- Converts **multiplexed 2D and 3D images from CellVoyager CV7000/8000** systems into OME-Zarr format, storing each acquisition as a separate OME-Zarr image in the same OME-Zarr plate.
+- Creates **OME-Zarr HCS plates**, combining all fields of view for each acquisition in a well into a single image.
+- Saves Fractal **region-of-interest (ROI) tables** for both individual fields of view and the entire well.
+- Handles overlapping fields of view by adjusting their positions to be non-overlapping, while preserving the original position data as additional columns in the ROI tables.
+
+### Limitations
+- This task currently does not support time-resolved data and ignores the time fields in CellVoyager metadata.