Skip to content

Commit 14b9f5f

Browse files
alistairewjalistairewj
authored
Merge pull request #1340 from MIT-LCP/mimic_iv_ed_v2_psql_fix
Update MIMIC-IV-ED psql build scripts to v2.0
2 parents 6cf9745 + 355993c commit 14b9f5f

File tree

12 files changed

+449
-56
lines changed

12 files changed

+449
-56
lines changed

‎.github/workflows/build-db.yml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,67 @@ jobs:
5353
psql -h $POSTGRES_HOST -U postgres -f mimic-iv/buildmimic/postgres/validate_demo.sql > validate_results.txt
5454
cat validate_results.txt
5555
56+
env:
57+
# The hostname used to communicate with the PostgreSQL service container
58+
POSTGRES_HOST: postgres
59+
PGPASSWORD: postgres
60+
# The default PostgreSQL port
61+
POSTGRES_PORT: 5432
62+
PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
63+
64+
build-mimic-iv-ed-psql:
65+
# Containers must run in Linux based operating systems
66+
runs-on: ubuntu-latest
67+
# Docker Hub image that `container-job` executes in
68+
container: node:latest
69+
70+
# Service containers to run with `container-job`
71+
services:
72+
# Label used to access the service container
73+
postgres:
74+
# Docker Hub image
75+
image: postgres
76+
# Provide the password for postgres
77+
env:
78+
POSTGRES_PASSWORD: postgres
79+
# Set health checks to wait until postgres has started
80+
options: >-
81+
--health-cmd pg_isready
82+
--health-interval 10s
83+
--health-timeout 5s
84+
--health-retries 5
85+
86+
steps:
87+
- name: Check out repository code
88+
uses: actions/checkout@v3
89+
90+
- name: Install psql command
91+
run: |
92+
apt-get update
93+
apt-get install --yes --no-install-recommends postgresql-client
94+
95+
- id: 'auth'
96+
uses: 'google-github-actions/auth@v0'
97+
with:
98+
project_id: ${{ secrets.GCP_PROJECT_ID }}
99+
credentials_json: ${{ secrets.GCP_SA_KEY }}
100+
101+
- name: 'Set up Cloud SDK'
102+
uses: 'google-github-actions/setup-gcloud@v0'
103+
104+
- name: Download demo and create tables on PostgreSQL
105+
run: |
106+
echo "Downloading MIMIC-IV demo from GCP."
107+
gsutil -q -u $PROJECT_ID -m cp -r gs://mimic-iv-archive/v2.0/demo/ed ./
108+
MIMIC_DATA_DIR=`pwd`/ed
109+
echo "Building and loading data into psql."
110+
cd mimic-iv-ed/buildmimic/postgres
111+
psql -q -h $POSTGRES_HOST -U postgres -f create.sql
112+
psql -q -h $POSTGRES_HOST -U postgres -v mimic_data_dir=$MIMIC_DATA_DIR -f load_gz.sql
113+
echo "Validating build."
114+
psql -h $POSTGRES_HOST -U postgres -f validate_demo.sql > validate_results.txt
115+
cat validate_results.txt
116+
56117
env:
57118
# The hostname used to communicate with the PostgreSQL service container
58119
POSTGRES_HOST: postgres

‎mimic-iv-ed/buildmimic/bigquery/schemas/ed/diagnosis.json‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,27 @@
1-
[{"name": "subject_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "stay_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "seq_num", "type": "INT64", "mode": "REQUIRED"}, {"name": "icd_code", "type": "STRING", "mode": "REQUIRED"}, {"name": "icd_title", "type": "STRING", "mode": "REQUIRED"}]
1+
[
2+
{
3+
"name": "subject_id",
4+
"type": "INT64",
5+
"mode": "REQUIRED"
6+
},
7+
{
8+
"name": "stay_id",
9+
"type": "INT64",
10+
"mode": "REQUIRED"
11+
},
12+
{
13+
"name": "seq_num",
14+
"type": "INT64",
15+
"mode": "REQUIRED"
16+
},
17+
{
18+
"name": "icd_code",
19+
"type": "STRING",
20+
"mode": "REQUIRED"
21+
},
22+
{
23+
"name": "icd_title",
24+
"type": "STRING",
25+
"mode": "REQUIRED"
26+
}
27+
]

‎mimic-iv-ed/buildmimic/bigquery/schemas/ed/edstays.json‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,47 @@
1-
[{"name": "subject_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "hadm_id", "type": "INT64", "mode": "NULLABLE"}, {"name": "stay_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "intime", "type": "DATETIME", "mode": "REQUIRED"}, {"name": "outtime", "type": "DATETIME", "mode": "REQUIRED"}]
1+
[
2+
{
3+
"name": "subject_id",
4+
"type": "INT64",
5+
"mode": "REQUIRED"
6+
},
7+
{
8+
"name": "hadm_id",
9+
"type": "INT64",
10+
"mode": "NULLABLE"
11+
},
12+
{
13+
"name": "stay_id",
14+
"type": "INT64",
15+
"mode": "REQUIRED"
16+
},
17+
{
18+
"name": "intime",
19+
"type": "DATETIME",
20+
"mode": "REQUIRED"
21+
},
22+
{
23+
"name": "outtime",
24+
"type": "DATETIME",
25+
"mode": "REQUIRED"
26+
},
27+
{
28+
"name": "gender",
29+
"type": "STRING",
30+
"mode": "REQUIRED"
31+
},
32+
{
33+
"name": "race",
34+
"type": "STRING",
35+
"mode": "NULLABLE"
36+
},
37+
{
38+
"name": "arrival_transport",
39+
"type": "STRING",
40+
"mode": "REQUIRED"
41+
},
42+
{
43+
"name": "disposition",
44+
"type": "STRING",
45+
"mode": "NULLABLE"
46+
}
47+
]

‎mimic-iv-ed/buildmimic/bigquery/schemas/ed/medrecon.json‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,47 @@
1-
[{"name": "subject_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "stay_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "charttime", "type": "DATETIME", "mode": "NULLABLE"}, {"name": "name", "type": "STRING", "mode": "NULLABLE"}, {"name": "gsn", "type": "STRING", "mode": "NULLABLE"}, {"name": "ndc", "type": "STRING", "mode": "NULLABLE"}, {"name": "etc_rn", "type": "INT64", "mode": "NULLABLE"}, {"name": "etccode", "type": "STRING", "mode": "NULLABLE"}, {"name": "etcdescription", "type": "STRING", "mode": "NULLABLE"}]
1+
[
2+
{
3+
"name": "subject_id",
4+
"type": "INT64",
5+
"mode": "REQUIRED"
6+
},
7+
{
8+
"name": "stay_id",
9+
"type": "INT64",
10+
"mode": "REQUIRED"
11+
},
12+
{
13+
"name": "charttime",
14+
"type": "DATETIME",
15+
"mode": "NULLABLE"
16+
},
17+
{
18+
"name": "name",
19+
"type": "STRING",
20+
"mode": "NULLABLE"
21+
},
22+
{
23+
"name": "gsn",
24+
"type": "STRING",
25+
"mode": "NULLABLE"
26+
},
27+
{
28+
"name": "ndc",
29+
"type": "STRING",
30+
"mode": "NULLABLE"
31+
},
32+
{
33+
"name": "etc_rn",
34+
"type": "INT64",
35+
"mode": "NULLABLE"
36+
},
37+
{
38+
"name": "etccode",
39+
"type": "STRING",
40+
"mode": "NULLABLE"
41+
},
42+
{
43+
"name": "etcdescription",
44+
"type": "STRING",
45+
"mode": "NULLABLE"
46+
}
47+
]

‎mimic-iv-ed/buildmimic/bigquery/schemas/ed/pyxis.json‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,37 @@
1-
[{"name": "subject_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "stay_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "charttime", "type": "DATETIME", "mode": "NULLABLE"}, {"name": "med_rn", "type": "INT64", "mode": "REQUIRED"}, {"name": "name", "type": "STRING", "mode": "NULLABLE"}, {"name": "gsn_rn", "type": "INT64", "mode": "REQUIRED"}, {"name": "gsn", "type": "STRING", "mode": "NULLABLE"}]
1+
[
2+
{
3+
"name": "subject_id",
4+
"type": "INT64",
5+
"mode": "REQUIRED"
6+
},
7+
{
8+
"name": "stay_id",
9+
"type": "INT64",
10+
"mode": "REQUIRED"
11+
},
12+
{
13+
"name": "charttime",
14+
"type": "DATETIME",
15+
"mode": "NULLABLE"
16+
},
17+
{
18+
"name": "med_rn",
19+
"type": "INT64",
20+
"mode": "REQUIRED"
21+
},
22+
{
23+
"name": "name",
24+
"type": "STRING",
25+
"mode": "NULLABLE"
26+
},
27+
{
28+
"name": "gsn_rn",
29+
"type": "INT64",
30+
"mode": "REQUIRED"
31+
},
32+
{
33+
"name": "gsn",
34+
"type": "STRING",
35+
"mode": "NULLABLE"
36+
}
37+
]

‎mimic-iv-ed/buildmimic/bigquery/schemas/ed/triage.json‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,57 @@
1-
[{"name": "subject_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "stay_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "temperature", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "heartrate", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "resprate", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "o2sat", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "sbp", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "dbp", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "pain", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "acuity", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "chiefcomplaint", "type": "STRING", "mode": "NULLABLE"}]
1+
[
2+
{
3+
"name": "subject_id",
4+
"type": "INT64",
5+
"mode": "REQUIRED"
6+
},
7+
{
8+
"name": "stay_id",
9+
"type": "INT64",
10+
"mode": "REQUIRED"
11+
},
12+
{
13+
"name": "temperature",
14+
"type": "NUMERIC",
15+
"mode": "NULLABLE"
16+
},
17+
{
18+
"name": "heartrate",
19+
"type": "NUMERIC",
20+
"mode": "NULLABLE"
21+
},
22+
{
23+
"name": "resprate",
24+
"type": "NUMERIC",
25+
"mode": "NULLABLE"
26+
},
27+
{
28+
"name": "o2sat",
29+
"type": "NUMERIC",
30+
"mode": "NULLABLE"
31+
},
32+
{
33+
"name": "sbp",
34+
"type": "NUMERIC",
35+
"mode": "NULLABLE"
36+
},
37+
{
38+
"name": "dbp",
39+
"type": "NUMERIC",
40+
"mode": "NULLABLE"
41+
},
42+
{
43+
"name": "pain",
44+
"type": "STRING",
45+
"mode": "NULLABLE"
46+
},
47+
{
48+
"name": "acuity",
49+
"type": "NUMERIC",
50+
"mode": "NULLABLE"
51+
},
52+
{
53+
"name": "chiefcomplaint",
54+
"type": "STRING",
55+
"mode": "NULLABLE"
56+
}
57+
]

‎mimic-iv-ed/buildmimic/bigquery/schemas/ed/vitalsign.json‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,57 @@
1-
[{"name": "subject_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "stay_id", "type": "INT64", "mode": "REQUIRED"}, {"name": "charttime", "type": "DATETIME", "mode": "NULLABLE"}, {"name": "temperature", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "heartrate", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "resprate", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "o2sat", "type": "NUMERIC", "mode": "NULLABLE"}, {"name": "sbp", "type": "INT64", "mode": "NULLABLE"}, {"name": "dbp", "type": "INT64", "mode": "NULLABLE"}, {"name": "rhythm", "type": "STRING", "mode": "NULLABLE"}, {"name": "pain", "type": "STRING", "mode": "NULLABLE"}]
1+
[
2+
{
3+
"name": "subject_id",
4+
"type": "INT64",
5+
"mode": "REQUIRED"
6+
},
7+
{
8+
"name": "stay_id",
9+
"type": "INT64",
10+
"mode": "REQUIRED"
11+
},
12+
{
13+
"name": "charttime",
14+
"type": "DATETIME",
15+
"mode": "NULLABLE"
16+
},
17+
{
18+
"name": "temperature",
19+
"type": "NUMERIC",
20+
"mode": "NULLABLE"
21+
},
22+
{
23+
"name": "heartrate",
24+
"type": "NUMERIC",
25+
"mode": "NULLABLE"
26+
},
27+
{
28+
"name": "resprate",
29+
"type": "NUMERIC",
30+
"mode": "NULLABLE"
31+
},
32+
{
33+
"name": "o2sat",
34+
"type": "NUMERIC",
35+
"mode": "NULLABLE"
36+
},
37+
{
38+
"name": "sbp",
39+
"type": "INT64",
40+
"mode": "NULLABLE"
41+
},
42+
{
43+
"name": "dbp",
44+
"type": "INT64",
45+
"mode": "NULLABLE"
46+
},
47+
{
48+
"name": "rhythm",
49+
"type": "STRING",
50+
"mode": "NULLABLE"
51+
},
52+
{
53+
"name": "pain",
54+
"type": "STRING",
55+
"mode": "NULLABLE"
56+
}
57+
]

‎mimic-iv-ed/buildmimic/bigquery/schemas/ed/vitalsign_hl7.json‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎mimic-iv-ed/buildmimic/postgres/README.md‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ After the database exists, the schema and tables can be created under this datab
4949
psql -d mimic -f create.sql
5050
```
5151

52-
Finally, loading the data into this data requires specifying the database name with `-d mimicived` again:
52+
Finally, loading the data into this data requires specifying the database name with `-d mimic` again:
5353

5454
```sh
5555
psql -d mimic -v ON_ERROR_STOP=1 -v mimic_data_dir=<INSERT MIMIC FILE PATH HERE> -f load.sql
@@ -93,10 +93,6 @@ NOTICE: table "XXXXXX" does not exist, skipping
9393

9494
This is normal. By default, the script attempts to delete tables before rebuilding them. If it cannot find the table to delete, it outputs a notice letting the user know.
9595

96-
## Older versions of PostgreSQL
97-
98-
If you have an older version of PostgreSQL, then it is still possible to load MIMIC, but modifications to the scripts are required. In particular, the scripts use declarative partitioning for larger tables to speed up queries. To read more about [declarative partitioning, see the PostgreSQL documentation](https://www.postgresql.org/docs/10/static/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE). You can remove declarative partitionining by modifying the create script, and removing it for each affected table. For example, chartevents in the `mimic_icu` schema uses declarative partitioning, and thus the create.sql script creates many partitions for chartevents: chartevents_01, chartevents_02, ..., etc. Replacing these with a single create statement for chartevents will make the script compatible for older versions of PostgreSQL.
99-
10096
### Other
10197

10298
Please see the [issues page](https://github.com/MIT-LCP/mimic-code/issues) to discuss other issues you may be having.

0 commit comments

Comments
 (0)