migrate getting started content

Author: alistairewj

jekyll-site/_docs/gettingstarted.md

@@ -0,0 +1,18 @@
+---
+title: "AWS"
+layout: default
+parent: "Cloud"
+grand_parent: "Getting Started"
+nav_order: 4
+description: "Currently MIMIC-III is available on AWS but MIMIC-IV is not. MIMIC-IV will be released on AWS in the future."
+---
+
+# AWS
+
+Recently, the MIT Laboratory of Computational Physiology (LCP) started hosting the MIMIC-III dataset on the AWS cloud through the AWS Public Dataset program. You can now use the MIMIC-III dataset via S3 without having to download, copy, or pay to store it. Instead, you can analyze the MIMIC-III dataset in the AWS Cloud using AWS services like Amazon EC2, Athena, AWS Lambda, or Amazon EMR. AWS Cloud availability enables quicker and cheaper research into the dataset.
+
+Services like Athena also offer you new analytical approaches to the MIMIC-III dataset. Using Athena, you can execute standard SQL queries against MIMIC-III without first loading the data into a database. Because you can reference the MIMIC-III dataset hosted by MIT LCP in Amazon S3, your analyses always reference the most recent version of the MIMIC-III dataset. Live hosting reduces upfront time and effort, eliminates data synchronization issues, improves data analysis, and reduces overall study costs.
+
+Once you have successfully requested access to MIMIC-III on AWS, you can follow the instructions linked below. These instructions initialize and execute an entire study performed on MIMIC-III using a hosted Jupyter notebook service on AWS.
+
+https://aws.amazon.com/blogs/big-data/perform-biomedical-informatics-without-a-database-using-mimic-iii-data-and-amazon-athena/

jekyll-site/_docs/gettingstarted/cloud/aws.md

@@ -0,0 +1,76 @@
+---
+title: "BigQuery"
+layout: default
+parent: "Cloud"
+grand_parent: "Getting Started"
+nav_order: 3
+description: "Access MIMIC IV on BigQuery."
+---
+
+# BigQuery
+
+BigQuery is a columnar, distributed relational database management system. BigQuery accesses only the columns specified in the query, making it ideal for data analysis workflows. [Read more about BigQuery in Google's cloud documentation](https://cloud.google.com/bigquery/).
+
+First, ensure you have been provisioned access to MIMIC III or IV on BigQuery. See the [cloud page for instructions](../../cloud). Once you have been provisioned access to using MIMIC on BigQuery, it's worthwhile to "pin" the dataset to see it on the BigQuery web tool.
+
+1. Go to the BigQuery console: http://console.cloud.google.com/bigquery
+2. If you haven't created a BigQuery project previously you will be asked to do so. You will need to enter information to pay for the cost of queries. For more details see: https://cloud.google.com/resource-manager/docs/creating-managing-projects
+3. On the left sidebar, next to "Explorer", click "+ ADD DATA". Click "Star a project by name".
+4. Type `physionet-data` and enter it in.
+5. In the sidebar on the left, you should now see the `physionet-data` project. Click the arrow to the left of `physionet-data` to expand the project.
+6. You should now see a number of datasets. Which datasets you see depends on the access provisioned to you.
+  * At a minimum, you will see the demo projects: `eicu_crd_demo` and `mimiciii_demo`.
+  * If you have successfully requested access to MIMIC-III, you will additionally see `mimiciii_clinical`, `mimiciii_demo`, `mimiciii_notes`, and `mimiciii_derived`.
+  * If you have successfully requested access to MIMIC-IV, you will additionally see `mimiciv_icu` and `mimiciv_hosp`.
+  * If you have successfully requested access to MIMIC-IV-ED, you will additionally see `mimiciv_ed`.
+  * If you have successfully requested access to MIMIC-IV-Note, you will additionally see `mimiciv_note`.
+  
+You are now ready to query the data! Try a simple query in the main dialogue box, while logged in under your project that pays for queries.
+
+```sql
+SELECT *
+FROM `physionet-data.mimiciv_hosp.patients`
+WHERE subject_id < 10000100
+ORDER BY subject_id
+```
+
+The query should return some data, and your browser window should be similar to the below:
+
+![Example output for the query](/img/cloud/bq/example_query.png)
+
+At this point you are ready to use MIMIC on BigQuery!
+
+A tutorial on using BigQuery to query MIMIC-III is available [here](/docs/iii/tutorials/intro-to-mimic-iii-bq).
+
+Note that we have a number of pre-generated "views" of the data. These are available in the `mimiciv_derived` dataset which you are free to query. All code used to generate these views has been made openly available on the [MIMIC-IV code repository](https://github.com/MIT-LCP/mimic-iv/).
+
+If you are having issues, see the [Troubleshooting section](#troubleshooting).
+
+## Troubleshooting
+
+### I get a pop-up about Terms of Service
+
+![Agree to the terms of service](/img/cloud/bq/agree_tos.png)
+
+You will need to agree to all GCP Terms of Service and adhere to their terms in order to use the data on BigQuery.
+
+### When I go to BigQuery, it asks me to create a project
+
+![Create a project on GCP](/img/cloud/bq/create_project.png)
+
+Almost all of your interactions with GCP are associated with a *project*. Importantly, all billing for your usage must be allotted to a single project.
+In order to use BigQuery you must have an activate project associated with your account. BigQuery offers a $300 free trial for first time users.
+
+Create a project and select it as your activate project. If you've done this correctly, then the top bar of the Google console page should stop saying "Select a project", and instead have your project name. For example, in the below, I have selected the project `alistairewj`, which is now the activate project:
+
+![Example of a working activate project](/img/cloud/bq/active_project.png)
+
+
+### I can only see `eicu_crd_demo` and `mimiciii_demo`
+
+These datasets are fully public, so the implication is that you have not been granted access to the full versions of the databases.
+Please (1) double check you have entered your cloud information into your PhysioNet profile, verifying any e-mails as needed, and (2) requested access to the specific cloud project on its respective PhysioNet project page.
+
+### I want to ask a question about MIMIC or raise an issue on the MIMIC Code Repository
+
+If none of the above have the answer, feel free to [raise an issue](https://github.com/MIT-LCP/mimic-code/issues) or [ask for advice](https://github.com/MIT-LCP/mimic-code/discussions) in the MIMIC repository.

jekyll-site/_docs/gettingstarted/cloud/bigquery.md

@@ -0,0 +1,39 @@
+---
+title: "Google Cloud Storage"
+layout: default
+parent: "Cloud"
+grand_parent: "Getting Started"
+nav_order: 5
+description: "Access the MIMIC datasets on Google Cloud Storage (GCS)."
+---
+
+# Google Cloud Storage
+
+The MIMIC databases are hosted on Google Cloud Storage (GCS). Access to the bucket, if available, can be requested via the PhysioNet project page. See [this page for more detail on this process](/docs/gettingstarted).
+
+Datasets available on GCS include:
+
+- [MIMIC-III](https://physionet.org/content/mimiciii/)
+- [MIMIC-IV](https://physionet.org/content/mimiciv/)
+- [MIMIC-IV-CXR](https://physionet.org/content/mimic-cxr/)\*\*
+- [MIMIC-IV-ED](https://physionet.org/content/mimic-iv-ed)
+
+
+\*\* MIMIC-IV-CXR is over 4.7 TB, almost entirely due to the size of the DICOMs.
+Users should strongly consider *not* downloading the data, and instead using it within Google Cloud Platform (GCP), which we support natively.
+GCP does not charge for data transfer within a region in GCP ([see this page for more details about network charges.](https://cloud.google.com/storage/pricing#network-pricing)).
+
+## Downloading a dataset using `gsutil`
+
+`gsutil` is a command-line tool for interacting with object stores.
+You'll need to install `gsutil` locally and authenticate with the same Google account you have linked to your PhysioNet account.
+
+See their instruction page for details on the install and configuration process: https://cloud.google.com/storage/docs/quickstart-gsutil
+
+Once you have `gsutil` installed and authenticated, you can download a dataset using the `gsutil -m cp` command, where `-m` requests multiprocessing. For example, you can download MIMIC-IV as follows:
+
+```
+gsutil -m cp -r gs://mimiciii-1.4.physionet.org ./
+```
+
+... which will download all the data (~7 GB) in the MIMIC-III project to your local folder.

jekyll-site/_docs/gettingstarted/cloud/gcs.md

@@ -0,0 +1,36 @@
+---
+title: "Cloud"
+layout: default
+parent: "Getting Started"
+has_children: true
+nav_order: 1
+description: "Access MIMIC on the Cloud"
+---
+
+# Cloud
+
+MIMIC-III and MIMIC-IV are made available via Physionet: [MIMIC-III](https://physionet.org/content/mimiciii/), [MIMIC-IV](https://physionet.org/content/mimiciv/). Beyond directly downloading the dataset from PhysioNet, there are a few mechanisms for accessing the data:
+
+* Accessing the data in BigQuery
+* Accessing the data on AWS (MIMIC-III)
+* Accessing the data in a Google Cloud storage bucket (MIMIC-IV)
+
+We **highly** recommend using MIMIC in BigQuery for the following reasons:
+
+* No setup required
+* Updates will be integrated into BigQuery when they are available
+* Derived concepts from the [MIMIC code repository](https://www.github.com/MIT-LCP/mimic-code) are precomputed and available on the `mimic_derived` dataset
+
+If absolutely necessary, it is possible to download the data from a Google cloud bucket; instructions are provided at the end of the document.
+Please do keep in mind that the PhysioNet team is covering the cost of downloading the dataset.
+
+## Accessing data on the cloud
+
+There are three steps to accessing data on the cloud:
+
+1. Link your cloud account to your PhysioNet profile
+2. Request access to the cloud resource
+3. Log-in to the appropriate service and navigate to the resource
+
+These steps assume you are already credentialed and have signed the data use agreement for MIMIC-IV.
+If you have not, read this page for instructions on gaining access to [MIMIC](/docs/gettingstarted).

jekyll-site/_docs/gettingstarted/cloud/index.md

@@ -0,0 +1,48 @@
+---
+title: "Linking your cloud account"
+layout: default
+parent: "Cloud"
+grand_parent: "Getting Started"
+nav_order: 1
+description: "How to connect your PhysioNet account to a cloud account."
+---
+
+# Linking your cloud account
+
+MIMIC is available for use via two cloud platforms: Google Cloud Platform (GCP) and Amazon Web Services (AWS). Access to these services is directly controlled via your PhysioNet account.
+
+We will assume you are a credentialed user on PhysioNet and have signed the MIMIC data use agreement.
+
+## Adding cloud credentials
+
+Go to your PhysioNet profile page.
+
+![Profile page on PhysioNet](/img/cloud/profile.png)
+
+Click "Emails":
+
+![Navigate to the e-mails page](/img/cloud/emails.png)
+
+For GCP access, ensure that one of your e-mails is a Google account. This can either be a gmail account (as in the picture), or a G Suite account if your organization is a member of G Suite. You can add an e-mail at the bottom of the page:
+
+![Navigate to the Cloud page](/img/cloud/add_email.png)
+
+You will need to verify your e-mail address before continuing (note: e-mail addresses are only used for GCP access, and not for AWS access).
+
+Once you have a verified e-mail address ready, navigate to the "Cloud" page on PhysioNet.
+
+![Navigate to the Cloud page](/img/cloud/cloud_page.png)
+
+You should see two options on this page: one for GCP, and one for AWS.
+
+![Profile cloud credentials](/img/cloud/credentials.png)
+
+For GCP, click the drop down menu and set your GCP e-mail to the Google account you provided in the earlier step.
+
+For AWS, add your AWS canonical ID. This is *not your e-mail*. It is a numeric identifier that can be found in your AWS cloud profile. [Click here to go to your AWS profile page](https://console.aws.amazon.com/billing/home?#/account). Then look for your "Account Id":
+
+![AWS ID](/img/cloud/aws/aws_id.png)
+
+## Linked account
+
+Once you have successfully linked a cloud account to your PhysioNet profile, the next step is to request access to [MIMIC](/docs/gettingstarted/cloud/request) for your cloud account.

jekyll-site/_docs/gettingstarted/cloud/link.md

@@ -0,0 +1,53 @@
+---
+title: "Accessing MIMIC-IV on the cloud"
+layout: default
+parent: "Cloud"
+grand_parent: "Getting Started"
+nav_order: 2
+description: "How to grant your linked cloud account access to MIMIC."
+---
+
+# Accessing MIMIC-IV on the cloud
+
+Now that your cloud credentials are available in PhysioNet, you can request access to databases within those cloud systems.
+Cloud access to PhysioNet projects such as MIMIC-IV and MIMIC-III are managed independently. You must request access to the cloud systems via their project pages (access is provisioned instantly for credentialed users who have signed the DUA).
+
+For MIMIC-III, go to the [MIMIC-III PhysioNet project page](https://physionet.org/content/mimiciii).
+For MIMIC-IV, go to the [MIMIC-IV PhysioNet project page](https://physionet.org/content/mimiciv).
+
+Once there, scroll to the bottom to the "Files" section.
+*If* the page shows a restricted-access warning, you need to get access to [MIMIC](/docs/gettingstarted).
+Otherwise, you should see the following:
+
+![Methods for accessing MIMIC-IV](/img/cloud/mimiciv_files.png)
+
+The following describes the access options listed above in the order they are listed:
+
+1. Download the ZIP file
+    * This downloads the data directly from the PhysioNet servers.
+2. Request access using Google BigQuery (**Cloud**)
+    * This option adds the Google e-mail in your PhysioNet account to a BigQuery access list
+    * This is required in order to use the data in BigQuery.
+3. Adds your AWS account ID to the access list for AWS (currently only available for MIMIC-III)(**Cloud**).
+    * This is necessary in order to access the data via AWS services. For information on how to use AWS, we [recommend reading this tutorial](https://aws.amazon.com/blogs/big-data/perform-biomedical-informatics-without-a-database-using-mimic-iii-data-and-amazon-athena/).
+4. Request access to the files using Google Cloud Storage Browser (only available for MIMIC-IV)(**Cloud**)
+    * This option adds your Google e-mail in your PhysioNet account to the GCP access list
+    * This is required in order to download the data from a storage bucket on GCP.
+5. Download the files using your terminal
+    * Provides a command for downloading the data from PhysioNet as individual CSV files using `wget` (when compared to the image above, your command will have a distinct username).
+
+For example, if you are interested in accessing MIMIC-IV on BigQuery, you would click "Request access using Google BigQuery". The page should provide you a green notification indicating you have been provided access.
+
+![Access granted to Google Cloud Platform's BigQuery service](/img/cloud/bq_provisioned.png)
+
+You will receive an e-mail detailing instructions for how to access MIMIC on BigQuery. Alternatively, instructions are also provided on the [BigQuery page](../bigquery).
+
+## Using data on the cloud
+
+Once you have been granted access to a cloud resource, the next step is to navigate to that resource in the cloud.
+
+* For the GCP Storage Bucket, click the link e-mailed to you.
+* For BigQuery, see the [BigQuery page](../bigquery). You may also be interested in the [querying tutorial on BigQuery] for [MIMIC-III](/docs/iii/tutorials/intro-to-mimic-iii-bq//) or [MIMIC-IV](/docs/iv/tutorials/bigquery/).
+* AWS access is currently unavailable for MIMIC-IV, but planned.
+
+Once you have access to MIMIC, we highly recommend you read the respective database introduction: [MIMIC-III](/docs/iii), [MIMIC-IV](/docs/iv/).