Merge pull request #1240 from armando-fandango/sqlite-import-for-mimic4

sqlite import for mimic4 as per #1052

Author: alistairewj

mimic-iv/buildmimic/sqlite/README.md

@@ -0,0 +1,71 @@
+# Building the MIMIC database with SQLite
+
+Either `import.sh` or `import.py` can be used to generate a [SQLite]([https://sqlite.org/index.html) database file from the MIMIC-IV demo or full dataset.
+
+`import.sh` is a shell script that will work with any POSIX compliant shell.
+It is memory efficient and does not require loading entire data files
+into memory. It only needs three things to run:
+
+1. A POSIX compliant shell (e.g., dash, bash, zsh, ksh, etc.)
+2. [SQLite]([https://sqlite.org/index.html)
+3. gzip (which is installed by default on any Linux/BSD/Mac variant)
+
+**Note:** The `import.sh` script will set all data fields to *text*.
+
+`import.py` is a python script. It requires the following to run:
+
+1. Python 3 installed
+2. SQLite
+3. [pandas](https://pandas.pydata.org/)
+
+## Step 1: Download the CSV or CSV.GZ files.
+
+- Download the MIMIC-IV dataset from: https://physionet.org/content/mimiciv/
+- Place `import.sh` or `import.py` into the same folder as the `csv` or `csv.gz` files
+
+i.e. your folder structure should resemble:
+
+```
+path/to/mimic-iv/
+├── import.sh
+├── import.py
+├── hosp
+│   ├── admissions.csv.gz
+│   ├── ...
+│   └── transfers.csv.gz
+└── hosp
+    ├── chartevents.csv.gz
+    ├── ...
+    └── procedureevents.csv.gz
+
+
+## Step 2: Edit the script if needed.
+
+`import.sh` does **not** need edits to work with either the demo or full dataset.
+Please continue to Step 3.
+
+If you are using the `import.py` script,
+it may be necessary to make minor edits to the `import.py` script. For example:
+
+- If your files are `.csv` rather than `csv.gz`, you will need to change `csv.gz` to `csv`.
+
+## Step 3: Generate the SQLite file
+
+To generate the SQLite file:
+
+If you are using `import.sh`, run on the command-line:
+
+```
+$ ./import.sh
+```
+
+If you are using `import.py`, run on the command-line:
+
+```
+$ python import.py
+```
+
+If loading the full dataset, this will take some time,
+particularly the `CHARTEVENTS` table.
+
+The scripts will ultimately generate an SQLite database file called `mimic4.db`.

mimic-iv/buildmimic/sqlite/import.py

@@ -0,0 +1,30 @@
+import os
+import sys
+
+from glob import glob
+import pandas as pd
+
+DATABASE_NAME = "mimic4.db"
+THRESHOLD_SIZE = 5 * 10**7
+CHUNKSIZE = 10**6
+CONNECTION_STRING = "sqlite:///{}".format(DATABASE_NAME)
+
+if os.path.exists(DATABASE_NAME):
+    msg = "File {} already exists.".format(DATABASE_NAME)
+    print(msg)
+    sys.exit()
+
+for f in glob("**/*.csv*", recursive=True):
+    print("Starting processing {}".format(f))
+    folder, filename = os.path.split(f)
+    tablename = filename.strip(".gz").strip(".csv").lower()
+    if os.path.getsize(f) < THRESHOLD_SIZE:
+        df = pd.read_csv(f)
+        df.to_sql(tablename, CONNECTION_STRING)
+    else:
+        # If the file is too large, let's do the work in chunks
+        for chunk in pd.read_csv(f, chunksize=CHUNKSIZE, low_memory=False):
+            chunk.to_sql(tablename, CONNECTION_STRING, if_exists="append")
+    print("Finished processing {}".format(f))
+
+print("Should be all done!")

mimic-iv/buildmimic/sqlite/import.sh

@@ -0,0 +1,51 @@
+#!/bin/sh
+
+# Copyright (c) 2021 Thomas Ward <[email protected]>
+# 
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+OUTFILE=mimic4.db
+
+if [ -s "$OUTFILE" ]; then
+    echo "File \"$OUTFILE\" already exists." >&2
+    exit 111
+fi
+
+for FILE in */**.csv*; do
+      # skip loop if glob didn't match an actual file
+      [ -f "$FILE" ] || continue
+      # trim off extension and lowercase file stem (e.g., HELLO.csv -> hello)
+      TABLE_NAME=$(echo "${FILE%%.*}" | tr "[:upper:]" "[:lower:]")
+      case "$FILE" in
+          *csv)
+              IMPORT_CMD=".import $FILE $TABLE_NAME"
+          ;;
+          # need to decompress csv before load
+          *csv.gz)
+              IMPORT_CMD=".import \"|gzip -dc $FILE\" $TABLE_NAME"
+          ;;
+          # not a data file so skip
+          *)
+              continue
+          ;;
+      esac
+      echo "Loading $FILE."
+      sqlite3 $OUTFILE <<EOF
+.headers on
+.mode csv
+$IMPORT_CMD
+EOF
+    echo "Finished loading $FILE."
+done
+
+echo "Finished loading data into $OUTFILE."