Skip to content

Commit 9c68ff9

Browse files
davehorsfalldavehorsfall
authored
Merge pull request #119 from haniffalab/dev
Create release 0.4.1
2 parents 0880964 + 29a271a commit 9c68ff9

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

55 files changed

+1911
-117
lines changed

‎.github/workflows/build-sphinx.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: build-docs
2+
3+
on:
4+
push:
5+
branches: [dev]
6+
7+
pull_request:
8+
branches: [main, dev]
9+
10+
jobs:
11+
run:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Checkout
15+
uses: actions/checkout@v2
16+
- name: Set up Python 3.10
17+
uses: actions/setup-python@v2
18+
with:
19+
python-version: "3.10"
20+
- name: Install dependencies
21+
run: |
22+
python -m pip install --upgrade pip
23+
pip install -r ./envs/requirements.txt
24+
pip install -r ./envs/dev/requirements.txt
25+
pip install -r ./envs/build_config/requirements.txt
26+
- name: Sphinx Build
27+
working-directory: ./sphinx
28+
run: |
29+
make html
30+
touch _build/html/.nojekyll

.github/workflows/sphinx-build.yml renamed to .github/workflows/deploy-sphinx.yml

Lines changed: 9 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,9 @@
1-
name: docs
1+
name: deploy-docs
22

33
on:
44
push:
5-
branches: [ main, dev ]
6-
tags: [ 'v*' ]
7-
8-
pull_request:
9-
branches: [ main, dev ]
5+
branches: [main]
6+
tags: ["v*"]
107

118
jobs:
129
run:
@@ -17,21 +14,22 @@ jobs:
1714
- name: Set up Python 3.10
1815
uses: actions/setup-python@v2
1916
with:
20-
python-version: '3.10'
17+
python-version: "3.10"
2118
- name: Install dependencies
2219
run: |
2320
python -m pip install --upgrade pip
2421
pip install -r ./envs/requirements.txt
25-
pip install sphinx sphinx-rtd-theme
22+
pip install -r ./envs/dev/requirements.txt
23+
pip install -r ./envs/build_config/requirements.txt
2624
- name: Sphinx Build
2725
working-directory: ./sphinx
28-
run: |
26+
run: |
2927
make html
3028
touch _build/html/.nojekyll
3129
- name: Deploy to GitHub Pages
3230
uses: JamesIves/github-pages-deploy-action@v4.4.1
3331
with:
34-
branch: dev
32+
branch: gh-pages
3533
folder: sphinx/_build/html
3634
target-folder: docs
37-
clean: true
35+
clean: true

‎.github/workflows/docker-builds.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: docker-builds
2+
3+
on:
4+
push:
5+
tags: ["v*"]
6+
7+
jobs:
8+
build:
9+
runs-on: ubuntu-latest
10+
steps:
11+
- name: Checkout
12+
uses: actions/checkout@v2
13+
- name: Get tag
14+
uses: oprypin/find-latest-tag@v1
15+
with:
16+
repository: haniffalab/webatlas-pipeline
17+
releases-only: true
18+
id: find-latest-tag
19+
- name: Set version
20+
run: |
21+
# TAG=${{ steps.find-latest-tag.outputs.tag }}
22+
TAG=${GITHUB_REF#refs/*/}
23+
VERSION=${TAG#v}
24+
echo Version: $VERSION
25+
echo "VERSION=$VERSION" >> $GITHUB_ENV
26+
- name: Login to Docker Hub
27+
uses: docker/login-action@v3
28+
with:
29+
username: ${{ secrets.DOCKERHUB_USERNAME }}
30+
password: ${{ secrets.DOCKERHUB_TOKEN }}
31+
- name: Build Docker (webatlas-pipeline)
32+
working-directory: ./envs
33+
run: |
34+
docker build --platform=linux/amd64 -t haniffalab/webatlas-pipeline:${VERSION} -f ./Dockerfile .
35+
- name: Build Docker (webatlas-pipeline-build-config)
36+
working-directory: ./envs/build_config
37+
run: |
38+
docker build --platform=linux/amd64 -t haniffalab/webatlas-pipeline-build-config:${VERSION} -f ./Dockerfile .
39+
- name: Push Docker images
40+
run: |
41+
docker push haniffalab/webatlas-pipeline:${VERSION}
42+
docker push haniffalab/webatlas-pipeline-build-config:${VERSION}

‎.github/workflows/tests-python.yml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,10 @@ name: python-tests
22

33
on:
44
push:
5-
branches: [ main, dev ]
6-
5+
branches: [main, dev]
6+
77
pull_request:
8-
branches: [ main, dev ]
8+
branches: [main, dev]
99

1010
jobs:
1111
run:
@@ -20,7 +20,7 @@ jobs:
2020
- name: Set up Python 3.10
2121
uses: actions/setup-python@v2
2222
with:
23-
python-version: '3.10'
23+
python-version: "3.10"
2424
- name: Clone ome-zarr-metadata
2525
uses: actions/checkout@v2
2626
with:
@@ -31,10 +31,11 @@ jobs:
3131
run: |
3232
python -m pip install --upgrade pip
3333
pip install -r ./envs/requirements.txt
34+
pip install -r ./envs/dev/requirements.txt
3435
cd ./ome-zarr-metadata && pre-commit install && pip install -e . && cd ../
3536
- name: Run tests
3637
run: python -m pytest --cov=bin tests/test_class.py
3738
env:
3839
PYTHONPATH: ./bin
3940
- name: Upload coverage to Codecov
40-
uses: codecov/codecov-action@v3
41+
uses: codecov/codecov-action@v3

‎bin/build_config_multimodal.py‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -38,14 +38,10 @@ def write_json(
3838
Args:
3939
project (str, optional): Project name. Defaults to "".
4040
datasets (dict[str, dict[str]], optional): Dictionary of datasets.
41-
Expected structure: { dataset_name: {
42-
"file_paths" : [],
43-
"images": {"raw": [], "label": []},
44-
"options": {},
45-
"obs_type": "cell"
46-
"is_spatial": True // if has images should be enough
47-
}
48-
}
41+
Expected structure: { dataset_name: { "file_paths" : [],
42+
"images": {"raw": [], "label": []},
43+
"options": {}, "obs_type": "cell",
44+
"is_spatial": True } }
4945
Defaults to {}.
5046
extended_features (Union[list[str], str], optional): List of features or
5147
string of single feature on which the expression matrix was extended
@@ -62,7 +58,6 @@ def write_json(
6258
outdir (str, optional): Directory in which the config file will be written to.
6359
Defaults to "./".
6460
"""
65-
6661
config = VitessceConfig(
6762
"1.0.15",
6863
name=str(title) if len(title) else str(project),

‎docs/_sources/index.rst.txt‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.7405818.svg
1010
:target: https://doi.org/10.5281/zenodo.7405818
1111

12-
WebAtlas Pipeline
12+
WebAtlas pipeline
1313
=================
1414

1515
This Nextflow pipeline processes spatial and single-cell experiment data for visualisation in `WebAtlas App`_.
@@ -38,6 +38,15 @@ Indices and tables
3838
* :ref:`modindex`
3939
* :ref:`search`
4040

41+
.. toctree::
42+
:maxdepth: 2
43+
:caption: Multimodal
44+
45+
multimodal/overview
46+
multimodal/configuration
47+
multimodal/run
48+
multimodal/visualise
49+
4150
.. toctree::
4251
:maxdepth: 2
4352
:hidden:

‎docs/_sources/modules.rst.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,6 @@ Modules
3737

3838
.. automodule:: integrate_image
3939
:members:
40+
41+
.. automodule:: build_config_multimodal
42+
:members:

‎docs/_sources/multimodal/configuration.rst.txt‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
.. _multimodal_configuration:
2+
3+
######################
4+
Multimodal configuration
5+
######################
6+
7+
After running the main conversion pipeline you can populate the required YAML parameters file to run the multimodal integration pipeline.
8+
9+
.. _multimodal_parameters_file:
10+
11+
***************
12+
Parameters file
13+
***************
14+
15+
The parameters file looks like this:
16+
17+
.. code-block:: yaml
18+
19+
outdir: "/path/to/output/"
20+
21+
url: http://localhost:3000/
22+
project: my_project
23+
title: "My Project"
24+
25+
data:
26+
-
27+
dataset: scrnaseq
28+
obs_type: cell
29+
anndata: /path/to/main/output/scrnaseq-anndata.zarr
30+
offset: 0
31+
is_spatial: false
32+
vitessce_options:
33+
spatial:
34+
xy: obsm/spatial
35+
mappings:
36+
obsm/X_umap: [0,1]
37+
matrix: X
38+
-
39+
dataset: visium
40+
obs_type: spot
41+
anndata: /path/to/main/output/visium-anndata.zarr
42+
offset: 1000000
43+
is_spatial: true
44+
raw_image: /path/to/main/output/visium-raw.zarr
45+
label_image: /path/to/main/output/visium-label.zarr
46+
vitessce_options:
47+
spatial:
48+
xy: obsm/spatial
49+
matrix: X
50+
51+
In contrast to the main conversion pipeline's parameters file, this file includes a single `project` to which multiple `datasets` belong to.
52+
53+
Each ``dataset`` block defines the name of the dataset and paths to the converted data and image files (if any).
54+
55+
Each ``dataset`` also requires a set of ``vitessce_options`` that specify the location of certain data (spatial coordinates, embeddings, expression matrix, etc.) within the AnnData object that is processed/generated.
56+
This follows the same structure as in the :ref:`main conversion's vitessce_options <vitessce_options>`.
57+
58+
Additionally, each ``dataset`` requires:
59+
60+
* ``obs_type``, the type of observation of the dataset. For example, "cell" or "spot".
61+
* ``offset``, an integer offset to add to the dataset's ID's so they don't clash with the other datasets.
62+
* ``is_spatial``, whether the dataset contains spatial information and has associated image files (raw and/or label images)
63+
64+
Given that raw images are only read but not modified the pipeline does not generate new output from them.
65+
In order for the output directory (defined by ``outdir``) to contain all necessary files that need to be served for the web application to consume,
66+
by default, the pipeline copies the raw images to the output directory.
67+
This process can take a long time depending on the size of the image.
68+
You may want to manually copy or move the image or serve it from its own directory separate from the rest of the output.
69+
The default copying can be disabled by setting ``copy_raw: false`` as a project-wide parameter (at the same level as ``outdir``, ``project``, etc).
70+
For example,
71+
72+
.. code-block:: yaml
73+
74+
outdir: "/path/to/output/"
75+
url: http://localhost:3000/
76+
project: my_project
77+
title: "My Project"
78+
copy_raw: false
79+
80+
81+
With additional features
82+
========================
83+
84+
Using the above example parameters file to run the multimodal integration pipeline will run the reindexing and intersection steps.
85+
To perform the concatenation of additional features (like celltypes) to visualise them as continuous values, some extra parameters need to be added.
86+
87+
As a project-wide parameter (at the same level as ``outdir``, ``project``, etc.):
88+
89+
* ``extend_feature_name``, the name of the additional feature. For example, "celltype"
90+
91+
And at a ``dataset`` level:
92+
93+
* ``extend_feature``, the location of the additional feature information.
94+
This can be either the path to a *cell2location* output file, or the location within the AnnData object where the feature is stored as a categorical within ``obs``.
95+
For example, ``/path/to/c2l.h5ad`` containing predicted continuous values, or ``obs/celltype`` containing categoricals.
96+
97+
The full parameters file will then look like this
98+
99+
.. code-block:: yaml
100+
101+
outdir: "/path/to/output/"
102+
103+
url: http://localhost:3000/
104+
project: my_project
105+
title: "My Project"
106+
107+
extend_feature_name: celltype
108+
109+
data:
110+
-
111+
dataset: scrnaseq
112+
obs_type: cell
113+
anndata: /path/to/main/output/scrnaseq-anndata.zarr
114+
extend_feature: obs/celltype
115+
offset: 0
116+
is_spatial: false
117+
vitessce_options:
118+
spatial:
119+
xy: obsm/spatial
120+
mappings:
121+
obsm/X_umap: [0,1]
122+
matrix: X
123+
-
124+
dataset: visium
125+
obs_type: spot
126+
anndata: /path/to/main/output/visium-anndata.zarr
127+
extend_feature: /path/to/c2l.h5ad
128+
offset: 1000000
129+
is_spatial: true
130+
raw_image: /path/to/main/output/visium-raw.zarr
131+
label_image: /path/to/main/output/visium-label.zarr
132+
vitessce_options:
133+
spatial:
134+
xy: obsm/spatial
135+
matrix: X
136+
137+
With this parameters the multimodal integration pipeline will concatenate the expression matrix with the additional feature values so both can be queried and visualised across datasets within the same portal.

‎docs/_sources/multimodal/overview.rst.txt‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
.. _multimodal_overview:
2+
3+
###################
4+
Multimodal overview
5+
###################
6+
7+
After the ``main.nf`` pipeline has been successfully run, WebAtlas can optionally process a group of multimodal datasets that
8+
share common features. This step will prepare the unified multimodal visualision for the web app.
9+
10+
The data outputs generated by running the ``main.nf`` conversion pipeline serve as inputs for this multimodal integration pipeline.
11+
12+
***************
13+
Running the multimodal pipeline
14+
***************
15+
16+
Follow the instructions below to run the multimodal pipeline.
17+
18+
1. :ref:`Configure <multimodal_configuration>` the parameters file for the ``multimodal.nf`` pipeline
19+
2. :ref:`Run <multimodal_run>` the ``multimodal.nf`` pipeline
20+
3. :ref:`Visualise <multimodal_visualise>` the multimodal data in a web browser
21+
22+
***************
23+
Tasks completed by the pipeline
24+
***************
25+
26+
The multimodal integration pipeline performs several tasks:
27+
28+
1. Reindex each dataset by a user-inputed offset so ID's do not clash between modalities.
29+
2. *Optionally*, concatenate other observation-by-feature matrices or categorical values to the expression matrix to enable their visualisation as continuous values. For example, a celltype prediction matrix and/or celltype categories.
30+
3. Find the intersection of features between all datasets and subset them to visualise only the intersection (as including features not present in all datasets can produce misleading visualisations.)
31+
**Note** the features are intersected using their index in the AnnData objects (``var`` table). All datasets must use the same type of data as index for the intersection to be correctly computed. For example, all datasets use names as index, or all datasets use IDs as index.

0 commit comments

Comments
 (0)