You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
alt="Click to view the tool presentation video on youtube">
63
+
<br>
64
+
<span> Click to view video tutorial</span>
38
65
</a>
39
66
</p>
40
67
41
68
### Objectives
42
-
The precision toxicology metadata manager is a tool created to help consortium partners of the [PrecisionTox](https://precisiontox.org/) who produce
43
-
data to create, find, validate and share metadata about samples they collect in the lab. The idea behind the tool is to
44
-
operate at planning stage, before organisms are even exposed to compounds and samples are collected. These samples are meant to be shipped
45
-
to a central partner who will perform material extractions for *RNAseq* and *mass-spectrometry* data acquisitions.
46
-
The tool ensures that metadata do not contain any error, can be used to find physical samples in the boxes shipped to the core facility and that the experimental results can be produced in
47
-
a FAIR and publishable way.
69
+
The precision toxicology metadata manager was created to help partners producing data in the
70
+
[PrecisionTox Consortium](https://precisiontox.org/) to create, find, validate and share metadata
71
+
about samples they collect in the lab. The idea behind the tool is to operate at planning stage, before organisms are
72
+
even exposed to compounds and samples are collected. These samples are meant to be shipped to a central partner who
73
+
will perform material extractions for *RNAseq* and *mass-spectrometry* data acquisitions. The tool ensures that
74
+
metadata do not contain any error, can be used to find physical samples in the boxes shipped to the core facility
75
+
and that the experimental results can be produced in a FAIR and publishable way.
48
76
<br>
49
77
Consortium partners producing samples are invited to fill a form based on an experimental design defined by the consortium.
50
-
The form generates excel files containing metadata about sample exposition and collection divided into
78
+
The form generates Excel files containing metadata about sample exposition and collection divided into
51
79
two sheets:
52
80
- a first sheet containing specific information about the samples. This includes which replicates are exposed
53
81
to which compound, at which dose, with which vehicle, and after how long they were collected. It also includes a unique
@@ -75,42 +103,6 @@ and unique identifiers for each record.
75
103
76
104
<imgsrc="./docs/source/_static/img/user_story.png"alt="Metadata pipeline for sample exposure and collection"style="max-width:700px; margin:auto; display:block;"/>
77
105
78
-
79
-
### Components:
80
-
81
-
#### The frontend client
82
-
A NuxtJS web application accessible at https://ptmm.netlify.app.
83
-
It is responsible for the user interface and the communication with the API.
84
-
85
-
#### The backend API
86
-
It is hosted in this repository. It contains a [Flask](https://flask.palletsprojects.com/en/2.3.x/) application exposing a REST API and is plugged to a relational
87
-
database through [SQLAlchemy](https://www.sqlalchemy.org/).
88
-
89
-
It is responsible for authentication, all functionalities logic and the persistence of
90
-
(meta)-data. It provides a Swagger documentation accessible at https://pretox.isa-tools.org/apidocs. The documentation
91
-
describes the API usage and provides a way to run queries through a web UI. The code is documented using ``docstrings``
92
-
and the documentation is available on ``readthedocs``. The application is entirely unit-tested, typehints are checked
93
-
with ``mypy``, code quality is surveyed by ``Codacy`` and styles are enforced by ``flake8``, all as part of the
94
-
continuous integration pipeline.
95
-
<br> <br>
96
-
The application source code is contained in the ``ptmd`` directory and divided as such:
97
-
- The ``api`` directory contains the flask application exposing the REST API. It includes routes definitions, the
98
-
JSON Web Token authentication logic and the validation of user inputs through JSON Schema.
99
-
- The ``boot`` directory contains the code responsible for booting the application, like seeding the initial data into
100
-
the database.
101
-
- The ``const`` directory contains the constants used throughout the application.
102
-
- The ``database`` directory contains the database models and complex queries. Interactions with the database is mostly
103
-
defined as methods of the model classes.
104
-
- The ``lib`` directory contains the code responsible for the business logic, like the interactions with the spreadsheets
105
-
and the Google Drive API, sending emails and generating ISA-JSON files.
106
-
- The ``resources`` directory contains the assets used by the application, like JSON schemas, swagger yaml files, data
107
-
files for organisations and chemicals, etc.
108
-
109
-
Tests are contained in the ``tests`` directory and divided mirroring the application exact structure. They require no
110
-
data files and no interaction with the database to be executed.
111
-
112
-
113
-
114
106
## Getting started
115
107
### Requirements:
116
108
- Python 3.9 or newer
@@ -126,6 +118,7 @@ python -m venv venv
126
118
source venv/bin/activate
127
119
pip install -r requirements.txt # for production
128
120
pip install -r requirements-dev.txt # for development
121
+
mv ptmd/resources/.env.example ptmd/resources/.env # Create the .env file
129
122
```
130
123
131
124
Before running the application you need to obtain a Google Drive ID and secret: head to your Google Cloud console and
@@ -156,32 +149,78 @@ The environment variables are divided into three categories:
156
149
-`SQLALCHEMY_DATABASE_URL`: this is the URL to your database. You can use sqlite, postgres, mysql, etc., e.g.
157
150
sqlite:///ptmd.db, and will need to change this before running the application.
158
151
-`SQLALCHEMY_SECRET_KEY`: this is the secret key used to encrypt hashes and other sensitive data.
159
-
-`SITE_URL`: the URL of the site.
152
+
-`SITE_URL`: the URL of the JavaScript client (used for sending activation and reset links through emails).
160
153
- The Google Drive API configuration variables:
161
154
-`GOOGLE_DRIVE_CLIENT_ID`: the Google Drive client ID obtainable in your Google Cloud Console (see above).
162
155
-`GOOGLE_DRIVE_CLIENT_SECRET`: the Google Drive client secret obtaining in your Google Cloud Console (see above).
163
156
-`GOOGLE_DRIVE_CREDENTIALS_FILEPATH`: the path where to store the credentials file created during first boot. This needs to
164
157
be a json file (e.g. credentials.json).
165
-
-`GOOGLE_DRIVE_SETTINGS_FILEPATH`: the path where to store the settings file created during first seed_db
166
-
e.g. settings.yaml) Not that a `.yml` extension <b>will not be accepted</b>, please use `.yaml` instead.
158
+
-`GOOGLE_DRIVE_SETTINGS_FILEPATH`: the path where to store the settings file created during first initialization
159
+
(e.g. settings.yaml). Note that a `.yml` extension <b>will not be accepted</b>, please use `.yaml` instead.
167
160
- The admin account configuration variables.
168
161
-`ADMIN_EMAIL`: the email address of the admin user. This is used to send emails to the admin user when a new user
169
162
registers.
170
163
-`ADMIN_USERNAME`: the username of the admin user. This is used to create the first admin user. Cannot be changed.
171
164
-`ADMIN_PASSWORD`: the password of the admin user. This is used to create the first admin user. Can be changed later.
172
165
173
-
You can now run the following command and accept the application in your browser. This is done once only during first seed_db.
166
+
You can now run the following command and accept the application in your browser. This is done once only during first initialization.
174
167
It will download the Google API credentials file and generate the database once you are done.
175
-
Finally, it will seed_db the flask API.
168
+
Finally, it will boot the flask API in your local host.
169
+
170
+
<u>Note</u>: The Google credentials can be generated on any machine and then copied to the server where the application is
171
+
running if path in the `.env` file matches the path on the server.
172
+
176
173
```shell
177
174
python -m app
178
175
```
179
176
180
177
Once the API is booted go to http://localhost:5000/apidocs to see the Swagger documentation.
In order to run the application in production you will need a web server and a WSGI. It has been tested with
181
+
Apache/Passenger and Nginx/Gunicorn. You will also want to configure your web server so that files under
182
+
the `ptmd/resources/` directory are served.
183
+
Finally, you need to set the publishing status of your application status to 'In Production' in your Google Cloud
184
+
Console. This will allow the application to refresh authentication tokens automatically.
184
185
186
+
## Architecture
187
+
188
+
### The frontend client
189
+
A NuxtJS web application accessible in [development](https://ptmm.netlify.app)
190
+
and [production](https://mm.precisiontox.org).
191
+
It is responsible for the user interface and the communication with the API.
192
+
193
+
### The backend API
194
+
It is hosted in this repository. It contains a [Flask](https://flask.palletsprojects.com/en/2.3.x/) application exposing a REST API and is plugged to a relational
195
+
database through [SQLAlchemy](https://www.sqlalchemy.org/).
196
+
197
+
### Development process
198
+
It is responsible for authentication, all functionalities logic and the persistence of
199
+
(meta)-data. It provides a [Swagger documentation](http://mmapi.precisiontox.org/apidocs/) describing the API usage
200
+
and enabling to build and run queries through a web UI. The code is documented using ``docstrings``
201
+
and the documentation is available on [readTheDocs](https://pretox-metadata-manager.readthedocs.io/en/latest/?badge=latest).
202
+
The application is entirely unit-tested, typehints are checked with ``mypy``, code quality is surveyed by ``Codacy``
203
+
and styles are enforced by ``flake8``, all as part of the continuous integration pipeline.
204
+
205
+
#### Components
206
+
The application source code is contained in the ``ptmd`` directory and divided as such:
207
+
- The ``api`` directory contains the flask application exposing the REST API. It includes routes definitions, the
208
+
JSON Web Token authentication logic and the validation of user inputs through JSON Schema.
209
+
- The ``boot`` directory contains the code responsible for booting the application, like seeding the initial data into
210
+
the database.
211
+
- The ``const`` directory contains the constants used throughout the application.
212
+
- The ``database`` directory contains the database models and complex queries. Interactions with the database is mostly
213
+
defined as methods of the model classes.
214
+
- The ``lib`` directory contains the code responsible for the business logic, like the interactions with the spreadsheets
215
+
and the Google Drive API, sending emails and generating ISA-JSON files.
216
+
- The ``resources`` directory contains the assets used by the application, like JSON schemas, swagger yaml files, data
217
+
files for organisations and chemicals, etc.
218
+
219
+
Tests are contained in the ``tests`` directory and divided mirroring the application exact structure. They require no
220
+
data files and no interaction with the database to be executed.
0 commit comments