-
Notifications
You must be signed in to change notification settings - Fork 0
📝 D3b-2196 document connecting to Hosted Airflow #9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
9c937b3
📝 D3b-2196 document connecting to mwaa
chris-s-friedman b8daab0
Update docs/guides/connect-to-airflow.md
chris-s-friedman 302bc4d
Update docs/guides/connect-to-hosted-infrastructure.md
chris-s-friedman 2a84248
Update docs/guides/connect-to-hosted-infrastructure.md
chris-s-friedman 8be7ca1
Update docs/guides/connect-to-airflow.md
chris-s-friedman File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # How to connect to the hosted Airflow instance | ||
|
|
||
| This guide documents how to connect to the AWS hosted Airflow instance | ||
|
|
||
| ## First Time Setup | ||
|
|
||
| For first time setup, see the instructions [here](./connect-to-hosted-infrastructure.md). | ||
| If you already completed this setup to connect to some other infrastructure in | ||
| the INCLUDE AWS account, such as the warehouse, you do not need to complete | ||
| the first time setup again. | ||
|
|
||
| ## Connecting to Airflow | ||
|
|
||
| 1. run the command `saml2aws login`. When using this tool, you will want to create a | ||
| profile for the AWS account `373997854230`. | ||
| 2. Navigate to the `ssm_forward_to_mwaa` directory within the `devops_utils` | ||
| repository you cloned on to your machine during first time setup | ||
| 3. Run the script `./mwaa_connect.sh include-mwaa-prd-airflow prd 443 us-east-1` | ||
| This script should should that you can now connect to airflow | ||
| 4. Open your web browser and navigate to google. Click on the `google apps` menu | ||
| to the left of your user profile picture at the right of the page and then | ||
| scroll down to and click on `Amazon Web Services`. You will then be presented | ||
| with a page to choose which aws account and role you would like to log into. | ||
| Choose the same one you chose in step 1 above. | ||
| 5. In your web browser, navigate to [https://localhost](https://localhost). This | ||
| should display an airflow login page. Login and then the page should now show | ||
| airflow's main page. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # How to connect to hosted infrastructure | ||
|
|
||
| The tools used to run and deploy dbt models are hosted in AWS and require setup | ||
| to use. This guide describes the steps needed to connect to hosted | ||
| infrastructure the first time, as well as what is needed to authenticate after | ||
| setup. | ||
|
|
||
| ## First Time Setup | ||
|
|
||
| ### Pre-requisites: Security, Complianze, and Permissions | ||
|
|
||
| Before connecting to the INCLUDE data warehouse, you must perform a couple of | ||
| actions to get an acount with permissions in the INCLUDE data warehouse. | ||
|
|
||
| First, you must request that your aws account have access to the INCLUDE AWS | ||
| account. At this time, this access is controlled by the D3B center's Devops | ||
| team. To request access, please reach out to @uklineale (Neel Kuila) via slack. | ||
| He will walk you through the process of getting access to the INCLUDE AWS | ||
| account and any compliance activities you need to complete. Please note that | ||
| the process to gain access may take a few days to complete. | ||
|
|
||
| Second, you must request that an account be created for you in the INCLUDE data | ||
| warehouse. To request an account, please reach out to your program's | ||
| administrator(s), requesting that an account be created for you. The | ||
| administrator(s) will create an account for you and add your account to the | ||
| appropriate role(s) within the warehouse, so that you will have access to the | ||
| appropriate tables immediately. | ||
|
|
||
| After this step is completed you will be furnished with database credentials as | ||
| well as the hostname of the database to connect to. | ||
|
|
||
| ### Setup SAML CLI access | ||
|
|
||
| You will connect to the warehouse using a command line interface (CLI) that will | ||
| authenticate that you can have access to the warehouse. To configure your access | ||
| you will use a CLI program from AWS, called `saml2aws`. | ||
|
|
||
| Instructions for installing saml2aws and using this CLI are [here](https://www.notion.so/d3b/Setup-SAML-Login-1056131f1200806ba182f7b7c1793a40?source=copy_link#18a6131f12008082af91f427d2bd71fa) | ||
| in the section `CLI Access`. Complete all steps in this section. When using | ||
| this tool, you will want to connect create a profile for the AWS account | ||
| `373997854230`. | ||
|
|
||
| ### Clone `devops_utils` repository | ||
|
|
||
| The [`devops_utils` repository](https://github.com/d3b-center/devops-utils) has | ||
| many scripts and tools for interacting with AWS resources. Clone this repository | ||
| to your machine. | ||
|
|
||
| ## Authentication | ||
|
|
||
| To authenticate your connection to aws resources, you will use the `saml2aws` | ||
| CLI described [above](#setup-saml-cli-access), using the command | ||
| `saml2aws login`. When using this tool, you will want to connect to a | ||
| profile for the AWS account `373997854230`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was able to successfully follow all the steps up until this one, where I got an unexpected error. I'm going to ask devops about it, but with that in mind I think it would be cool if we could add a screenshot that shows what this final page should look like!