Skip to content

Commit e3364b8

Browse files
GahhhGahhh
committed
fix bugs
Update README.md
1 parent a68676c commit e3364b8

File tree

8 files changed

+86
-108
lines changed

8 files changed

+86
-108
lines changed

‎README.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
![Flask](https://img.shields.io/badge/Flask-2.3.x-orange)
1515

1616
<p>
17-
This project aims to create a secure and easy-to-use mechanism for managing the already acquired and to-be-generated clinical trial data into a central searchable service, which can be used to analyse clinical data and create deep-learning models for predicting various features of interest. The learning system provides a RESTful API and token-based authentication for ease of integration with existing and new applications that produce or require access to the de-identified patient data to researchers, clinicians, and other health professionals who want to use the available clinical data.
17+
This project aims to create a secure and easy-to-use mechanism for managing the already acquired and to-be-generated clinical trial data into a central searchable service, which can be used to analyse clinical data and create deep-learning models for predicting various features of interest. The learning system provides a RESTful API for ease of integration with existing and new applications that produce or require access to the de-identified patient data to researchers, clinicians, and other health professionals who want to use the available clinical data.
1818
</p>
1919

2020
<p align="center">
@@ -38,16 +38,22 @@ This project aims to create a secure and easy-to-use mechanism for managing the
3838
![Architecture](docs/deployment_guide/images/architecture.png)
3939

4040
## Installation
41-
The project contains three main components:
41+
The project contains two main components:
4242
* The database server ([data_service](src/data_service))
4343
- The main flask application that hosts the database server.
44-
* The web application ([admin_console](src/admin_console))
44+
* The database front application ([Learndb_manager](https://github.com/Image-X-Institute/learndb-manager))
45+
- The database management application based on Reactjs.
46+
47+
48+
49+
The fowllowing components are not in use anymore, but still be able to run:
50+
* The web application ([admin_console](src/admin_console)) - Archive
4551
- The web application for admin to manage the import data.
46-
* The content uploader application ([content_uploader](src/content_uploader))
52+
* The content uploader application ([content_uploader](src/content_uploader)) - Archive
4753
- The application for users to import data into the database.
4854

4955
The documentation for the project can be found here:
50-
* The user guide for general users can be found in the [docs](docs/User_Guide_Data_import.pdf) folder.
56+
* The user guide for general users can be found in the [docs](docs/Docs_for_importing_new_patient_data.pdf) folder.
5157
* The documentation for the project can be found in the [docsrc](docs/deployment_guide) folder.
5258

5359
The file structure of the project is as follows:

‎docs/Docs for importing new patient data.pdf

638 KB
Binary file not shown.

‎docs/trial_folder_structure/VALKIM.json

Lines changed: 11 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -1,65 +1,28 @@
11
{
22
"prescription":{
33
"rt_ct_pres": {
4-
"path":"/{clinical_trial}/{test_centre}/Patient Files/PAT{centre_patient_no}/",
4+
"path":"/{clinical_trial}/{test_centre}/PlanningFiles/Patient{centre_patient_no}/",
55
"display_name":"Pre-treatment CT series",
66
"field_type":"folder",
77
"level": "prescription",
88
"allowed":["application/dicom"]
9-
},
10-
"rt_dose_pres": {
11-
"path":"/{clinical_trial}/{test_centre}/Patient Files/PAT{centre_patient_no}/",
12-
"display_name":"RT Dose DICOM",
13-
"field_type":"folder",
14-
"level": "prescription",
15-
"allowed":["application/dicom"]
16-
},
17-
"rt_mri_pres": {
18-
"path":"/{clinical_trial}/{test_centre}/Patient Files/PAT{centre_patient_no}/",
19-
"display_name":"Pre-treatment MRI series",
20-
"field_type":"folder",
21-
"level": "prescription",
22-
"allowed":["application/dicom"]
23-
},
24-
"rt_plan_pres": {
25-
"path":"/{clinical_trial}/{test_centre}/Patient Files/PAT{centre_patient_no}/",
26-
"display_name":"RT Plan DICOM",
27-
"field_type":"folder",
28-
"level": "prescription",
29-
"allowed":["application/dicom"]
30-
},
9+
},
3110
"rt_structure_pres": {
32-
"path":"/{clinical_trial}/{test_centre}/Patient Files/PAT{centre_patient_no}/",
11+
"path":"/{clinical_trial}/{test_centre}/PlanningFiles/Patient{centre_patient_no}/",
3312
"display_name":"RT Structure set DICOM",
3413
"field_type":"folder",
3514
"level": "prescription",
3615
"allowed":["application/dicom"]
37-
},
38-
"planned_dvh_pres": {
39-
"path":"/{clinical_trial}/{test_centre}/Patient Files/PAT{centre_patient_no}/",
40-
"display_name":"Cumulative DVH",
41-
"field_type":"folder",
42-
"level": "prescription",
43-
"allowed":["text/plain"]
44-
},
16+
},
4517
"centroid_pres":{
46-
"path":"/{clinical_trial}/{test_centre}/Centroid Files/PAT{centre_patient_no}/",
47-
"display_name":"GTV Isocentre offset",
18+
"path":"/{clinical_trial}/{test_centre}/Centroid files/Patient{centre_patient_no}/",
19+
"display_name":"Centroid files",
4820
"field_type":"folder",
4921
"level": "prescription",
5022
"allowed":["text/plain"]
5123
},
52-
"magik_visual":{
53-
"path":"/{clinical_trial}/{test_centre}/MAGIKmodel/PAT{centre_patient_no}/",
54-
"display_name":"MAGIK_visualisation",
55-
"field_type":"folder",
56-
"level": "prescription",
57-
"allowed":["image/gif",
58-
"video/avi",
59-
"application/octet-stream"]
60-
},
6124
"magik_model":{
62-
"path":"/{clinical_trial}/{test_centre}/MAGIKmodel/PAT{centre_patient_no}/MAGIK_ModelFiles/",
25+
"path":"/{clinical_trial}/{test_centre}/MAGIKmodel/Patient{centre_patient_no}/",
6326
"display_name":"MAGIK_model",
6427
"field_type":"folder",
6528
"level": "prescription",
@@ -69,7 +32,7 @@
6932
"application/octet-stream"]
7033
},
7134
"marker_offsets":{
72-
"path":"/{clinical_trial}/{test_centre}/Marker Offsets/PAT{centre_patient_no}/",
35+
"path":"/{clinical_trial}/{test_centre}/Marker offsets/Patient{centre_patient_no}/",
7336
"display_name":"Marker offsets",
7437
"field_type":"folder",
7538
"level": "prescription",
@@ -82,63 +45,16 @@
8245
},
8346
"fraction":{
8447
"kv_images_path": {
85-
"path":"/{clinical_trial}/{test_centre}/Treatment Files/PAT{centre_patient_no}/",
48+
"path":"/{clinical_trial}/{test_centre}/Treatment Files/Patient{centre_patient_no}/",
8649
"display_name":"kV Images",
8750
"field_type":"folder",
8851
"level": "fraction",
8952
"allowed":["image/tiff",
9053
"application/dicom",
9154
"application/octet-stream"]
92-
},
93-
"trajectory_logs_path": {
94-
"path":"/{clinical_trial}/{test_centre}/Marker Trajectory/PAT{centre_patient_no}/",
95-
"display_name":"Trajectory Log Files",
96-
"field_type":"folder",
97-
"level": "fraction",
98-
"allowed":["application/octet-stream",
99-
"application/zip",
100-
"application/x-7z-compressed",
101-
"application/gzip",
102-
"application/x-bzip2",
103-
"application/x-bzip"]
104-
},
105-
"dvh_track_path": {
106-
"path":"/{clinical_trial}/{test_centre}/Dose Reconstruction/DVH/PAT{centre_patient_no}/",
107-
"display_name":"DVH plan (tracked)",
108-
"field_type":"folder",
109-
"level": "fraction",
110-
"allowed":["text/plain"]
111-
},
112-
"dvh_no_track_path": {
113-
"path":"/{clinical_trial}/{test_centre}/Dose Reconstruction/DVH/PAT{centre_patient_no}/",
114-
"display_name":"DVH plan (not tracked)",
115-
"field_type":"folder",
116-
"level": "fraction",
117-
"allowed":["text/plain"]
118-
},
119-
"dicom_track_plan_path": {
120-
"path":"/{clinical_trial}/{test_centre}/Dose Reconstruction/DICOM/PAT{centre_patient_no}/",
121-
"display_name":"DICOM plan for dose per fraction (tracked)",
122-
"field_type":"folder",
123-
"level": "fraction",
124-
"allowed":["application/dicom"]
125-
},
126-
"dicom_no_track_plan_path": {
127-
"path":"/{clinical_trial}/{test_centre}/Dose Reconstruction/DICOM/PAT{centre_patient_no}/",
128-
"display_name":"DICOM plan for dose per fraction (not tracked)",
129-
"field_type":"folder",
130-
"level": "fraction",
131-
"allowed":["application/dicom"]
132-
},
133-
"rpm_path": {
134-
"path":"/{clinical_trial}/{test_centre}/Treatment files/PAT{centre_patient_no}/",
135-
"display_name":"Surface monitoring trace",
136-
"field_type":"folder",
137-
"level": "fraction",
138-
"allowed":["text/plain", "text/csv"]
139-
},
55+
},
14056
"magik_logs":{
141-
"path":"/{clinical_trial}/{test_centre}/Markerless output/PAT{centre_patient_no}/",
57+
"path":"/{clinical_trial}/{test_centre}/Markerless output/Patient{centre_patient_no}/",
14258
"display_name":"MAGIK log files",
14359
"field_type":"folder",
14460
"level": "fraction",

‎scripts/service/adminConsole.service

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ User=learndb
99
Group=learndb
1010
WorkingDirectory=/home/learndb/git_repos/The-Real-Time-Imaging-Database/src/admin_console
1111
ExecStart=/home/learndb/git_repos/The-Real-Time-Imaging-Database/scripts/service/admin_service.sh
12+
StandardOutput=append:/datalog/admin_console_stdout.log
13+
StandardError=append:/datalog/admin_console_stderr.log
1214
ExecStop=pkill -u learndb adminApp.py
1315
Restart=always
1416
RestartSec=5

‎scripts/service/learndb.service

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ User=learndb
99
Group=learndb
1010
WorkingDirectory=/home/learndb/git_repos/The-Real-Time-Imaging-Database/src/data_service
1111
ExecStart=/home/learndb/git_repos/The-Real-Time-Imaging-Database/scripts/service/data_service.sh
12+
StandardOutput=append:/datalog/data_service_stdout.log
13+
StandardError=append:/datalog/data_service_stderr.log
1214
ExecStop=pkill -u learndb dataApp.py
1315
Restart=always
1416
RestartSec=10

‎src/data_service/ClinicalTrials.py

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
from AccessManager import accessManagerInstance, AccessType
1010
from sys import stderr
1111
import psycopg2.extras
12+
import re
1213

1314

1415
class ClinicalTrials:
@@ -183,17 +184,36 @@ def getEndpointData(self, endpoint:str,
183184
firstParam = True
184185
for param in paramsOfInterest:
185186
if param in requestParams:
187+
calculationSign = '='
188+
paramValue = requestParams[param]
189+
if re.search(r'[<>]', requestParams[param]):
190+
calculationSign = re.search(r'[<>]', requestParams[param]).group()
191+
paramValue = re.sub(r'[<>]', '', requestParams[param])
186192
if len(dbRelations) == 1 and firstParam:
187193
strQuery += " WHERE "
188194
else:
189195
strQuery += " AND "
190196

191197
if firstParam:
192198
firstParam = False
193-
194-
strQuery += paramsOfInterest[param]["table"] + "." \
195-
+ paramsOfInterest[param]["column"] + " = " \
196-
+ "'" + requestParams[param] + "'"
199+
# add or condition here, the keyword is -or-
200+
if re.search(r'-or-', paramValue):
201+
paramValueList = paramValue.split('-or-')
202+
strQuery += "("
203+
firstValue = True
204+
for value in paramValueList:
205+
if not firstValue:
206+
strQuery += " OR "
207+
else:
208+
firstValue = False
209+
strQuery += dbRelations[0]["table"] + "." \
210+
+ paramsOfInterest[param]["column"] + calculationSign \
211+
+ "'" + value + "'"
212+
strQuery += ")"
213+
else:
214+
strQuery += paramsOfInterest[param]["table"] + "." \
215+
+ paramsOfInterest[param]["column"] + calculationSign \
216+
+ "'" + paramValue + "'"
197217
strQuery += ";\n"
198218

199219
if config.APP_DEBUG_MODE:

‎src/data_service/gui/web_gui/apis.html

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -142,8 +142,12 @@ <h1 class="main_title">Clinical Database RESTful API Reference</h1>
142142
</tr>
143143
{% for param in apis[endpoint]['query_params'] %}
144144
<tr class="field"><td><strong>{{param}}</strong>: {{apis[endpoint]['query_params'][param]['description']}}</td></tr>
145-
{% set exampleUrl = endpoint+'?'+param+'='+apis[endpoint]['query_params'][param]['example'] %}
146-
<tr class="param-example"><td><em>Example: <a href="{{exampleUrl}}">{{exampleUrl}}</a></em> </td></tr>
145+
{% if apis[endpoint]['query_params'][param]['example'] == '' %}
146+
{% set exampleUrl = endpoint+'?'+apis[endpoint]['query_params'][param]['or_condition'] %}
147+
{% else %}
148+
{% set exampleUrl = endpoint+'?'+param+'='+apis[endpoint]['query_params'][param]['example'] %}
149+
{% endif %}
150+
<tr class="param-example"><td><em>Example: <a href="{{exampleUrl}}">{{exampleUrl}}</a></em> </td></tr>
147151
{% endfor %}
148152
<tr>
149153
<td>
@@ -177,4 +181,4 @@ <h1 class="main_title">Clinical Database RESTful API Reference</h1>
177181
in the form of an access token. To get access to a authorisation token,
178182
please <a href="apply-access">apply here</a>.
179183
</body>
180-
</html>
184+
</html>

‎src/data_service/resources/api_mapping.json

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,13 @@
152152
"column": "linac_type",
153153
"description": "Search by specifying the Manufacturer and model of the LINAC used",
154154
"example": "Varian Truebeam"
155+
},
156+
"or_condition": {
157+
"table":"",
158+
"column":"",
159+
"description": "Search parameters with or condition by using '-or-' keyword, Example: tumour_stie=liver-or-Liver",
160+
"example":"",
161+
"or_condition": "trial=LARK&tumour_site=liver-or-Liver"
155162
}
156163
},
157164
"required_fields":[
@@ -461,6 +468,19 @@
461468
"column": "imaging_ma",
462469
"description": "Search by specifying the imaging modality for MV",
463470
"example": "CBCT"
471+
},
472+
"tumour_site": {
473+
"table": "patient",
474+
"column": "tumour_site",
475+
"description": "Search by affected organ or anatomical region",
476+
"example": "prostate"
477+
},
478+
"or_condition": {
479+
"table":"",
480+
"column":"",
481+
"description": "Search parameters with or condition by using '-or-' keyword, Example: tumour_stie=liver-or-Liver",
482+
"example":"",
483+
"or_condition": "trial=LARK&tumour_site=liver-or-Liver"
464484
}
465485
},
466486
"required_fields":[
@@ -608,6 +628,14 @@
608628
"table": "fraction",
609629
"column": "imaging_ma"
610630
}
631+
},
632+
{
633+
"property": "tumour_site",
634+
"description": "The organ/region of the tumour being treated",
635+
"field": {
636+
"table": "patient",
637+
"column": "tumour_site"
638+
}
611639
}
612640
],
613641
"object_fields": [
@@ -1639,4 +1667,4 @@
16391667
}
16401668
]
16411669
}
1642-
}
1670+
}

0 commit comments

Comments
 (0)