Merge branch 'main' into add_image_normalization

Author: sophiamaedler

.pre-commit-config.yaml

@@ -20,7 +20,7 @@ repos:
               scportrait_data
           )
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.0
+    rev: v0.15.2
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix, --unsafe-fixes]
@@ -39,7 +39,7 @@ repos:
     rev: v1.19.1
     hooks:
       - id: mypy
-        args: [--no-strict-optional, --ignore-missing-imports]
+        args: [--python-version=3.10, --no-strict-optional, --ignore-missing-imports]
         additional_dependencies:
           ["types-requests", "types-attrs", "types-PyYAML"]
   - repo: local

docs/CONTRIBUTING.md

@@ -214,7 +214,7 @@ The docs for scPortrait are updated and built automatically whenever code is mer
 make clean
 make html
 ```
-4. open the file `scportriat/docs/_build/html/index.html` in your favorite browser
+4. open the file `scportrait/docs/_build/html/index.html` in your favorite browser
 
 ## Writing Tests
 
@@ -246,11 +246,40 @@ pytest tests/unit_tests --disable-warnings
 E2E tests run larger workflow scenarios to ensure that multiple components of the pipeline work correctly together (e.g., segmentation → extraction → featurization → selection).
 These tests are **much slower** and may require larger example data.
 
+If an E2E test depends on remotely downloaded data/config files, it must be annotated with `@pytest.mark.requires_dataset(...)`.
+This is required because scPortrait runs a preflight data check before executing E2E test bodies and exits early if required data cannot be downloaded, avoiding misleading downstream failures.
+See [Test Markers](#test-markers) for details and examples.
+
 Run only E2E tests:
 ```console
 pytest tests/e2e_tests --disable-warnings
 ```
 
+### Test Markers
+
+scPortrait uses pytest markers to describe test metadata and e2e data requirements.
+
+- `@pytest.mark.slow`: marks slow-running tests (used to identify expensive tests such as full-pipeline e2e runs).
+- `@pytest.mark.requires_dataset(...)`: declares which remote datasets/configs an e2e test needs.
+
+The e2e preflight fixture in `tests/e2e_tests/conftest.py` reads `requires_dataset(...)` markers from the selected tests and preloads only those datasets before test bodies execute. If preloading fails, pytest exits early and the remaining e2e tests are not executed.
+
+Example:
+
+```python
+@pytest.mark.slow
+@pytest.mark.requires_dataset("dataset_1_config", "test_dataset")
+def test_full_pipeline_e2e(tmp_path):
+    ...
+```
+
+Run marker-filtered tests:
+
+```console
+pytest -m slow
+pytest -m "not slow"
+```
+
 ### Running All Tests
 
 To run both unit and E2E tests together:

docs/Makefile

@@ -2,20 +2,21 @@
 #
 
 github:
-	@make html
+	@$(MAKE) html
 
 # You can set these variables from the command line, and also
 # from the environment for the first two.
 SPHINXOPTS    ?=
-SPHINXBUILD   ?= sphinx-build
+PYTHON        ?= python
+SPHINXBUILD   ?= $(PYTHON) -m sphinx.cmd.build
 SOURCEDIR     = .
 BUILDDIR      = ./_build
 
 # Put it first so that "make" without argument is like "make help".
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: help Makefile
+.PHONY: help clean github Makefile
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).

docs/conf.py

@@ -62,6 +62,7 @@ def setup(app):
     "auto_examples/**.py",
     "auto_examples/**.md5",
 ]
+suppress_warnings = ["config.cache"]
 
 # autodoc_mock_imports = []
 autodoc_mock_imports = []  # type: ignore

docs/index.rst

@@ -11,12 +11,12 @@ scPortrait is a scalable toolkit to analyse single-cell image datasets. This Pyt
 Installation
 =============
 
-Check out the installation instructions :ref:`here<installation>`. You can validate your installation by running one of the example notebooks `here <https://github.com/MannLabs/scPortrait-notebooks/tree/main/example_projects/example_1>`_.
+Check out the installation instructions :ref:`here<installation>`. You can validate your installation by running one of the `example notebooks <https://github.com/MannLabs/scPortrait-notebooks/tree/main/example_projects/example_1>`_.
 
 Getting Started
 ===============
 
-You can check out our :ref:`worfklow overview<workflow>` or our `in-depth tutorial <https://mannlabs.github.io/scPortrait/pages/notebooks/_notebook_workflow_walkthrough.html>`_ to get started with scPortrait. For more detailed information on the package, checkout the more detailed guides :ref:`here<detailed_guide>`. You can find a collection of example projects which run on datasets provided within scPortrait to help you get started with `here <https://github.com/MannLabs/scPortrait-notebooks/tree/main/example_projects>`_. If you encounter issues feel free to `open up a git issue <https://github.com/MannLabs/scPortrait/issues>`_.
+You can check out our :ref:`workflow overview<workflow>` or our `in-depth tutorial <https://mannlabs.github.io/scPortrait/pages/notebooks/_notebook_workflow_walkthrough.html>`_ to get started with scPortrait. For more detailed information on the package, checkout the more detailed guides :ref:`here<detailed_guide>`. You can find a collection of example projects which run on datasets provided within scPortrait in the `example projects repository <https://github.com/MannLabs/scPortrait-notebooks/tree/main/example_projects>`_. If you encounter issues feel free to `open up a git issue <https://github.com/MannLabs/scPortrait/issues>`_.
 
 Citing our Work
 ================
@@ -51,7 +51,6 @@ Documentation
 
    auto_examples/code_snippets/index
 
-..
    pages/example_notebooks.ipynb
 
 .. toctree::

docs/pages/module/processing.rst

@@ -11,3 +11,4 @@ masks
 ############
 .. automodule:: scportrait.processing.masks
     :members:
+    :no-index:

docs/pages/module/tools/ml.rst

@@ -39,6 +39,7 @@ plmodels
 .. autoclass:: scportrait.tools.ml.plmodels.MultilabelSupervisedModel
     :members:
     :show-inheritance: False
+    :no-index:
 
 pretrained_models
 #################

docs/pages/workflow.rst

@@ -39,7 +39,7 @@ Stitching
 During stitching, individual fields-of-view are registered and combined into a single whole-slide image. Using parallel processing and out-of-memory computation scPortrait can efficiently align and assemble large
 datasets that exceed the memory capacities of a single machine. The stitching process is optional and can be skipped if the input images are already in the form of a single whole-slide image or individual images need to be processed.
 
-You can find an example notebook illustrating the stitching of individual tif files to whole-slide images :ref:`here <example_stitching>`
+You can find an example notebook illustrating the stitching of individual tif files to whole-slide images :ref:`in the stitching tutorial <example_stitching>`
 
 .. _quickstart_segmentation:
 
@@ -124,7 +124,7 @@ During featurization, the extracted single cell images are passed to a phenotype
 
 The type of featurization applicable to your use case will depend on the type of data you are working with and the biological question you are trying to answer. For example, in our `first publication <https://doi.org/10.1101/2023.06.01.542416>`_ we describe a deep learning-based binary image classifier that identifies individual cells defective in a biological process called "autophagy". Multiple featurization runs can be performed on the same dataset so that different featurization approaches can be used in parallel.
 
-You can find more information on running an inference within a scPortrait Project in this `notebook <https://mannlabs.github.io/scPortrait/html/pages/notebooks/example_scPortrait_project.html#Classification-of-extracted-single-cells>`_.
+You can find more information on running an inference within a scPortrait Project in this `classification notebook <https://mannlabs.github.io/scPortrait/html/pages/notebooks/example_scPortrait_project.html#Classification-of-extracted-single-cells>`_.
 
 .. _quickstart_selection:
 
@@ -135,7 +135,7 @@ The selection step takes a list of cells that have been selected based on their
 
 During selection, the appearance of selected cells can be modified, e.g. by applying an erosion or dilation or by smoothing the shapes to further stream-line the excision process. Please refer to the `py-lmd <https://github.com/MannLabs/py-lmd>`_ library for more details on the available parameters.
 
-You can also find an example selection workflow outlined in this `notebook <https://mannlabs.github.io/scPortrait/html/pages/notebooks/example_scPortrait_project.html#Exporting-Cutting-contours-for-excision-on-a-Leica-LMD7>`_.
+You can also find an example selection workflow outlined in this `selection notebook <https://mannlabs.github.io/scPortrait/html/pages/notebooks/example_scPortrait_project.html#Exporting-Cutting-contours-for-excision-on-a-Leica-LMD7>`_.
 
 .. _detailed_guide:
 

docs/pages/workflow/segmentation_workflow.rst

@@ -498,7 +498,7 @@ If you utilize this segmentation workflow please also consider citing the `cellp
 Cytosol Only Cellpose segmentation
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-This segmentation workflow is also built around the cellular segmentation algorithm `cellpose <https://cellpose.readthedocs.io/en/latest/>`_  but only performs a cytosol segmentation. Unlike the :ref:`DAPI segmentation cellpose <_DAPI_segmentation_cellpose>` workflow it uses two input channels to generate a single output mask. The generated single cell datasets using this segmentation method will contain all signal from within the cytosolic region.
+This segmentation workflow is also built around the cellular segmentation algorithm `cellpose <https://cellpose.readthedocs.io/en/latest/>`_  but only performs a cytosol segmentation. Unlike the :ref:`DAPI segmentation cellpose <DAPI_segmentation_cellpose>` workflow it uses two input channels to generate a single output mask. The generated single cell datasets using this segmentation method will contain all signal from within the cytosolic region.
 
 As for the :ref:`cytosol segmentation cellpose <Cytosol_segmentation_cellpose>` workflow it is highly recommended to utilize a GPU. If your system has more than one GPU available, in a ShardedSegmentation context, you can specify the number of GPUs to be used via the configuration file (``nGPUs``).
 

pyproject.toml

@@ -130,3 +130,10 @@ show_error_codes = true
 show_error_context = true
 ignore_missing_imports = true
 no_strict_optional = true
+explicit_package_bases = true
+
+[tool.pytest.ini_options]
+markers = [
+    "slow: marks slow-running tests",
+    "requires_dataset(*names): e2e datasets/configs that must be preloaded before test execution",
+]