Skip to content

Commit c366aeb

Browse files
chris-s-friedmanchris-s-friedman
authored
Merge pull request #9 from include-dcc/docs/cf/d3b-2196-connecting-to-mwaa
📝 D3b-2196 document connecting to Hosted Airflow
2 parents 65ee12c + 8be7ca1 commit c366aeb

File tree

3 files changed

+86
-40
lines changed

3 files changed

+86
-40
lines changed

‎docs/guides/connect-to-airflow.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# How to connect to the hosted Airflow instance
2+
3+
This guide documents how to connect to the AWS hosted Airflow instance
4+
5+
## First Time Setup
6+
7+
For first time setup, see the instructions [here](./connect-to-hosted-infrastructure.md).
8+
If you already completed this setup to connect to some other infrastructure in
9+
the INCLUDE AWS account, such as the warehouse, you do not need to complete
10+
the first time setup again.
11+
12+
## Connecting to Airflow
13+
14+
1. run the command `saml2aws login`. When using this tool, you will want to create a
15+
profile for the AWS account `373997854230`.
16+
2. Navigate to the `ssm_forward_to_mwaa` directory within the `devops_utils`
17+
repository you cloned on to your machine during first time setup
18+
3. Run the script `./mwaa_connect.sh include-mwaa-prd-airflow prd 443 us-east-1`
19+
This script should should that you can now connect to airflow
20+
4. Open your web browser and navigate to google. Click on the `google apps` menu
21+
to the left of your user profile picture at the right of the page and then
22+
scroll down to and click on `Amazon Web Services`. You will then be presented
23+
with a page to choose which aws account and role you would like to log into.
24+
Choose the same one you chose in step 1 above.
25+
5. In your web browser, navigate to [https://localhost](https://localhost). This
26+
should display an airflow login page. Login and then the page should now show
27+
airflow's main page.

‎docs/guides/connect-to-hosted-infrastructure.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# How to connect to hosted infrastructure
2+
3+
The tools used to run and deploy dbt models are hosted in AWS and require setup
4+
to use. This guide describes the steps needed to connect to hosted
5+
infrastructure the first time, as well as what is needed to authenticate after
6+
setup.
7+
8+
## First Time Setup
9+
10+
### Pre-requisites: Security, Complianze, and Permissions
11+
12+
Before connecting to the INCLUDE data warehouse, you must perform a couple of
13+
actions to get an acount with permissions in the INCLUDE data warehouse.
14+
15+
First, you must request that your aws account have access to the INCLUDE AWS
16+
account. At this time, this access is controlled by the D3B center's Devops
17+
team. To request access, please reach out to @uklineale (Neel Kuila) via slack.
18+
He will walk you through the process of getting access to the INCLUDE AWS
19+
account and any compliance activities you need to complete. Please note that
20+
the process to gain access may take a few days to complete.
21+
22+
Second, you must request that an account be created for you in the INCLUDE data
23+
warehouse. To request an account, please reach out to your program's
24+
administrator(s), requesting that an account be created for you. The
25+
administrator(s) will create an account for you and add your account to the
26+
appropriate role(s) within the warehouse, so that you will have access to the
27+
appropriate tables immediately.
28+
29+
After this step is completed you will be furnished with database credentials as
30+
well as the hostname of the database to connect to.
31+
32+
### Setup SAML CLI access
33+
34+
You will connect to the warehouse using a command line interface (CLI) that will
35+
authenticate that you can have access to the warehouse. To configure your access
36+
you will use a CLI program from AWS, called `saml2aws`.
37+
38+
Instructions for installing saml2aws and using this CLI are [here](https://www.notion.so/d3b/Setup-SAML-Login-1056131f1200806ba182f7b7c1793a40?source=copy_link#18a6131f12008082af91f427d2bd71fa)
39+
in the section `CLI Access`. Complete all steps in this section. When using
40+
this tool, you will want to connect create a profile for the AWS account
41+
`373997854230`.
42+
43+
### Clone `devops_utils` repository
44+
45+
The [`devops_utils` repository](https://github.com/d3b-center/devops-utils) has
46+
many scripts and tools for interacting with AWS resources. Clone this repository
47+
to your machine.
48+
49+
## Authentication
50+
51+
To authenticate your connection to aws resources, you will use the `saml2aws`
52+
CLI described [above](#setup-saml-cli-access), using the command
53+
`saml2aws login`. When using this tool, you will want to connect to a
54+
profile for the AWS account `373997854230`.

‎docs/guides/connect-to-the-warehouse.md‎

Lines changed: 5 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -4,47 +4,12 @@ This guide documents how to connect to the include postgres data warehouse.
44

55
## First Time Setup
66

7-
### Pre-requisites: Security, Complianze, and Permissions
7+
For first time setup, see the instructions [here](./connect-to-hosted-infrastructure.md)
8+
If you already completed this setup to connect to some other infrastructure in
9+
the INCLUDE AWS account, such as Airflow, you do not need to complete
10+
the first time setup again.
811

9-
Before connecting to the INCLUDE data warehouse, you must perform a couple of
10-
actions to get an acount with permissions in the INCLUDE data warehouse.
11-
12-
First, you must request that your aws account have access to the INCLUDE AWS
13-
account. At this time, this access is controlled by the D3B center's Devops
14-
team. To request access, please reach out to @uklineale (Neel Kuila) via slack.
15-
He will walk you through the process of getting access to the INCLUDE AWS
16-
account and any compliance activities you need to complete. Please note that
17-
the process to gain access may take a few days to complete.
18-
19-
Second, you must request that an account be created for you in the INCLUDE data
20-
warehouse. To request an account, please reach out to your program's
21-
administrator(s), requesting that an account be created for you. The
22-
administrator(s) will create an account for you and add your account to the
23-
appropriate role(s) within the warehouse, so that you will have access to the
24-
appropriate tables immediately.
25-
26-
After this step is completed you will be furnished with database credentials as
27-
well as the hostname of the database to connect to.
28-
29-
### Setup SAML CLI access
30-
31-
You will connect to the warehouse using a command line interface (CLI) that will
32-
authenticate that you can have access to the warehouse. To configure your access
33-
you will use a CLI program from AWS, called `saml2aws`.
34-
35-
Instructions for installing saml2aws and using this CLI are [here](https://www.notion.so/d3b/Setup-SAML-Login-1056131f1200806ba182f7b7c1793a40?source=copy_link#18a6131f12008082af91f427d2bd71fa)
36-
in the section `CLI Access`. Complete all steps in this section. When using
37-
this tool, you will want to connect create a profile for the AWS account
38-
`373997854230`.
39-
40-
### Clone `devops_utils` repository
41-
42-
The [`devops_utils` repository](https://github.com/d3b-center/devops-utils) has
43-
many scripts and tools for interacting with AWS resources. Clone this repository
44-
to your machine and then navigate to the directory within the repository
45-
`ssm_forward_to_db`.
46-
47-
## Connecting to the warehouse after the first time
12+
## Connecting to the warehouse
4813

4914
1. run the command `saml2aws login`. When using this tool, you will want to connect create a
5015
profile for the AWS account `373997854230`.

0 commit comments

Comments
 (0)