Skip to content

Commit ee4cc66

Browse files
jluethijluethi
committed
Initial commit
0 parents  commit ee4cc66

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

77 files changed

+975
-0
lines changed

‎.copier-answers.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Do not edit - changes here will be overwritten by Copier
2+
_commit: v0.1.3
3+
_src_path: gh:fractal-analytics-platform/fractal-tasks-template
4+
author_email: [email protected]
5+
author_name: Joel Luethi
6+
package_name: fractal_helper_tasks
7+
project_name: fractal-helper-tasks
8+
project_short_description: Collection of Fractal helper tasks
9+
project_url: ''
10+

‎LICENSE‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
BSD 3-Clause License
2+
3+
Copyright (c) 2023, Joel Luethi
4+
5+
Redistribution and use in source and binary forms, with or without
6+
modification, are permitted provided that the following conditions are met:
7+
8+
1. Redistributions of source code must retain the above copyright notice, this
9+
list of conditions and the following disclaimer.
10+
11+
2. Redistributions in binary form must reproduce the above copyright notice,
12+
this list of conditions and the following disclaimer in the documentation
13+
and/or other materials provided with the distribution.
14+
15+
3. Neither the name of the copyright holder nor the names of its
16+
contributors may be used to endorse or promote products derived from
17+
this software without specific prior written permission.
18+
19+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
20+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
22+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
23+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
25+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
26+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
27+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎README.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# fractal-helper-tasks
2+
3+
Collection of Fractal helper tasks
4+
5+
## Development instructions
6+
7+
This instructions are only relevant *after* you completed both the `copier
8+
copy` command and the git/GitLab/GitHub initialization phase - see
9+
[README](https://github.com/fractal-analytics-platform/fractal-tasks-template#readme)
10+
for details.
11+
12+
1. It is recommended to work from an isolated Python virtual environment:
13+
```console
14+
# Create the virtual environment in the folder venv
15+
python -m venv venv
16+
# Activate the Python virtual environment
17+
source venv/bin/activate
18+
# Deactivate the virtual environment, when you don't need it any more
19+
deactivate
20+
```
21+
2. You can install your package locally as in:
22+
```console
23+
# Install only fractal_helper_tasks:
24+
python -m pip install -e .
25+
# Install both fractal_helper_tasks and development dependencies (e.g. pytest):
26+
python -m pip install -e ".[dev]"
27+
```
28+
29+
3. Enjoy developing the package.
30+
31+
4. The template already includes a sample task ("Thresholding Task"). Whenever
32+
you change its input parameters or docstring, re-run
33+
```console
34+
python src/fractal_helper_tasks/dev/update_manifest.py
35+
git add src/fractal_helper_tasks/__FRACTAL_MANIFEST__.json
36+
git commit -m'Update `__FRACTAL_MANIFEST__.json`'
37+
git push origin main
38+
```
39+
40+
5. If you add a new task, you should also add a new item to the `task_list`
41+
property in `src/fractal_helper_tasks/__FRACTAL_MANIFEST__.json`. A minimal example
42+
may look like
43+
```json
44+
{
45+
"name": "My Second Task",
46+
"executable": "my_second_task.py",
47+
"input_type": "zarr",
48+
"output_type": "zarr",
49+
"meta": {
50+
"some-property": "some-value"
51+
},
52+
}
53+
```
54+
Notes:
55+
56+
* After adding a task, you should also update the manifest (see point 4 above).
57+
* The minimal example above also includes the `meta` task property; this is optional, and you can remove it if it is not needed.
58+
59+
6. Run the test suite (with somewhat verbose logging) through
60+
```console
61+
python -m pytest --log-cli-level info -s
62+
```
63+
7. Build the package through
64+
```console
65+
python -m build
66+
```
67+
This command will create the release distribution files in the `dist` folder.
68+
The wheel one (ending with `.whl`) is the one you can use to collect your tasks
69+
within Fractal.

‎pyproject.toml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Project metadata (see https://peps.python.org/pep-0621)
2+
[project]
3+
name = "fractal-helper-tasks"
4+
version = "0.0.1"
5+
description = "Collection of Fractal helper tasks"
6+
readme = "README.md"
7+
license = { text = "BSD-3-Clause" }
8+
authors = [
9+
{ name = "Joel Luethi", email = "[email protected]" },
10+
]
11+
12+
# Required Python version and dependencies
13+
requires-python = ">=3.8"
14+
dependencies = ["fractal-tasks-core"]
15+
16+
# Optional dependencies (e.g. for `pip install -e ".[dev]"`, see
17+
# https://peps.python.org/pep-0621/#dependencies-optional-dependencies)
18+
[project.optional-dependencies]
19+
dev = ["devtools", "pytest", "requests", "build", "jsonschema"]
20+
21+
# Build options (see https://peps.python.org/pep-0517)
22+
[build-system]
23+
requires = ["setuptools"]
24+
build-backend = "setuptools.build_meta"
25+
26+
[tool.setuptools.packages.find]
27+
where = ["src"]
28+
include = ["fractal_helper_tasks"]
29+
30+
# Always include the __FRACTAL_MANIFEST__.json file in the package
31+
[tool.setuptools.package-data]
32+
"*" = ["__FRACTAL_MANIFEST__.json"]

‎src/fractal_helper_tasks/__FRACTAL_MANIFEST__.json‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
{
2+
"manifest_version": "1",
3+
"task_list": [
4+
{
5+
"name": "Thresholding Task",
6+
"executable": "thresholding_task.py",
7+
"input_type": "zarr",
8+
"output_type": "zarr",
9+
"args_schema": {
10+
"title": "ThresholdingTask",
11+
"type": "object",
12+
"properties": {
13+
"input_paths": {
14+
"title": "Input Paths",
15+
"type": "array",
16+
"items": {
17+
"type": "string"
18+
},
19+
"description": "Path to the parent folder of the NGFF image. This task only supports a single input path. (standard argument for Fractal tasks, managed by Fractal server)."
20+
},
21+
"output_path": {
22+
"title": "Output Path",
23+
"type": "string",
24+
"description": "This argument is not used in this task. (standard argument for Fractal tasks, managed by Fractal server)."
25+
},
26+
"component": {
27+
"title": "Component",
28+
"type": "string",
29+
"description": "Path of the NGFF image, relative to `input_paths[0]`. (standard argument for Fractal tasks, managed by Fractal server)."
30+
},
31+
"metadata": {
32+
"title": "Metadata",
33+
"type": "object",
34+
"description": "This argument is not used in this task. (standard argument for Fractal tasks, managed by Fractal server)."
35+
}
36+
},
37+
"required": [
38+
"input_paths",
39+
"output_path",
40+
"component",
41+
"metadata"
42+
],
43+
"additionalProperties": false
44+
},
45+
"docs_info": "Short description of thresholding_task.\n\nLong description of thresholding_task.",
46+
"docs_link": "https://example.com"
47+
}
48+
],
49+
"has_args_schemas": true,
50+
"args_schema_version": "pydantic_v1"
51+
}

‎src/fractal_helper_tasks/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
"""
2+
Collection of Fractal helper tasks
3+
"""

‎src/fractal_helper_tasks/dev/update_manifest.py‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
"""
2+
Script to generate JSON schemas for task arguments afresh, and write them
3+
to the package manifest.
4+
5+
This script is a copy of [the original one in
6+
fractal-tasks-core](https://github.com/fractal-analytics-platform/fractal-tasks-core/blob/main/fractal_tasks_core/dev/update_manifest.py),
7+
with minor changes.
8+
"""
9+
import json
10+
import logging
11+
from importlib import import_module
12+
from pathlib import Path
13+
14+
from fractal_tasks_core.dev.lib_args_schemas import (
15+
create_schema_for_single_task,
16+
)
17+
from fractal_tasks_core.dev.lib_task_docs import create_docs_info
18+
19+
20+
PACKAGE = "fractal_helper_tasks"
21+
22+
23+
if __name__ == "__main__":
24+
25+
# Read manifest
26+
imported_package = import_module(PACKAGE)
27+
manifest_path = (
28+
Path(imported_package.__file__).parent / "__FRACTAL_MANIFEST__.json"
29+
)
30+
with manifest_path.open("r") as f:
31+
manifest = json.load(f)
32+
33+
# Set global properties of manifest
34+
manifest["has_args_schemas"] = True
35+
manifest["args_schema_version"] = "pydantic_v1"
36+
37+
# Loop over tasks
38+
task_list = manifest["task_list"]
39+
for ind, task in enumerate(task_list):
40+
executable = task["executable"]
41+
logging.info(f"[{executable}] START")
42+
43+
# Create new JSON Schema for task arguments
44+
schema = create_schema_for_single_task(executable, package=PACKAGE)
45+
manifest["task_list"][ind]["args_schema"] = schema
46+
47+
# Update docs_info, based on task-function description
48+
docs_info = create_docs_info(executable, package=PACKAGE)
49+
docs_link = ""
50+
if docs_info:
51+
manifest["task_list"][ind]["docs_info"] = docs_info
52+
if docs_link:
53+
manifest["task_list"][ind]["docs_link"] = docs_link
54+
55+
logging.info(f"[{executable}] END (new schema/description/link)")
56+
print()
57+
58+
with manifest_path.open("w") as f:
59+
json.dump(manifest, f, indent=2)
60+
f.write("\n")
61+
logging.info(f"Up-to-date manifest stored in {manifest_path.as_posix()}")

‎src/fractal_helper_tasks/thresholding_task.py‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
"""
2+
This is the Python module for my_task
3+
"""
4+
5+
import logging
6+
from typing import Any
7+
from pathlib import Path
8+
9+
import zarr
10+
import dask.array as da
11+
12+
from pydantic.decorator import validate_arguments
13+
14+
from fractal_tasks_core.ngff import load_NgffImageMeta
15+
from fractal_tasks_core.pyramids import build_pyramid
16+
17+
@validate_arguments
18+
def thresholding_task(
19+
*,
20+
input_paths: list[str],
21+
output_path: str,
22+
component: str,
23+
metadata: dict[str, Any],
24+
) -> None:
25+
"""
26+
Short description of thresholding_task.
27+
28+
Long description of thresholding_task.
29+
30+
Args:
31+
input_paths: Path to the parent folder of the NGFF image.
32+
This task only supports a single input path.
33+
(standard argument for Fractal tasks, managed by Fractal server).
34+
output_path: This argument is not used in this task.
35+
(standard argument for Fractal tasks, managed by Fractal server).
36+
component: Path of the NGFF image, relative to `input_paths[0]`.
37+
(standard argument for Fractal tasks, managed by Fractal server).
38+
metadata: This argument is not used in this task.
39+
(standard argument for Fractal tasks, managed by Fractal server).
40+
"""
41+
42+
# Use the first of input_paths
43+
input_path = (Path(input_paths[0]) / component).as_posix()
44+
logging.info(f"input_path set to {input_path}")
45+
46+
# Parse and log several NGFF-image metadata attributes
47+
ngff_image_meta = load_NgffImageMeta(input_path)
48+
logging.info(f" Axes: {ngff_image_meta.axes_names}")
49+
logging.info(f" Number of pyramid levels: {ngff_image_meta.num_levels}")
50+
logging.info(f" Linear coarsening factor for YX axes: {ngff_image_meta.coarsening_xy}")
51+
logging.info(f" Full-resolution ZYX pixel sizes (micrometer): {ngff_image_meta.get_pixel_sizes_zyx(level=0)}")
52+
logging.info(f" Coarsening-level-1 ZYX pixel sizes (micrometer): {ngff_image_meta.get_pixel_sizes_zyx(level=1)}")
53+
54+
# Load the highest-resolution multiscale array through dask.array
55+
array_czyx = da.from_zarr(f"{input_path}/0")
56+
logging.info(f"{array_czyx=}")
57+
58+
# Set values below 100 to 0
59+
array_max = array_czyx.max().compute()
60+
array_min = array_czyx.min().compute()
61+
logging.info(f"Pre thresholding: {array_min=}, {array_max=}")
62+
array_czyx[array_czyx < 99] = 99
63+
array_czyx[array_czyx > 1000] = 1000
64+
array_max = array_czyx.max().compute()
65+
array_min = array_czyx.min().compute()
66+
logging.info(f"Post thresholding: {array_min=}, {array_max=}")
67+
68+
# Write the processed array back to the same full-resolution Zarr array
69+
array_czyx.to_zarr(f"{input_path}/0", overwrite=True)
70+
71+
# Starting from on-disk full-resolution data, build and write to disk a
72+
# pyramid of coarser levels
73+
build_pyramid(
74+
zarrurl=input_path,
75+
overwrite=True,
76+
num_levels=ngff_image_meta.num_levels,
77+
coarsening_xy=ngff_image_meta.coarsening_xy,
78+
)
79+
80+
81+
if __name__ == "__main__":
82+
from fractal_tasks_core.tasks._utils import run_fractal_task
83+
84+
run_fractal_task(task_function=thresholding_task)

‎tests/__init__.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
import fractal_helper_tasks
2+
import json
3+
from pathlib import Path
4+
5+
6+
PACKAGE = "fractal_helper_tasks"
7+
PACKAGE_DIR = Path(fractal_helper_tasks.__file__).parent
8+
MANIFEST_FILE = PACKAGE_DIR / "__FRACTAL_MANIFEST__.json"
9+
with MANIFEST_FILE.open("r") as f:
10+
MANIFEST = json.load(f)
11+
TASK_LIST = MANIFEST["task_list"]

0 commit comments

Comments
 (0)