|
| 1 | +--- |
| 2 | +description: Create an Ockam Redshift outlet node using Cloudformation template |
| 3 | +--- |
| 4 | + |
| 5 | +# Ockam Node for Amazon Redshift |
| 6 | + |
| 7 | +This guide contains instructions to launch within AWS environment, an  |
| 8 | + |
| 9 | +* An Ockam Redshift Outlet Node within an AWS environment |
| 10 | +* An Ockam Redshift Inlet Node: |
| 11 | + * Within an AWS environment, or |
| 12 | + * Using Docker in any environment |
| 13 | + |
| 14 | +The walkthrough demonstrates: |
| 15 | + |
| 16 | +1. Running an Ockam Redshift Outlet node in your AWS environment that contains a _private_ Amazon Redshift Serverless or Amazon Redshift Provisioned Cluster |
| 17 | +2. Setting up Ockam Redshift inlet nodes using either AWS or Docker from any location. |
| 18 | +3. Verifying secure communication between Redshift clients and Amazon Redshift Database. |
| 19 | + |
| 20 | +Read: “[How does Ockam work?](https://docs.ockam.io/how-does-ockam-work)” to learn about end-to-end trust establishment. |
| 21 | + |
| 22 | +<figure><img src="../../../../.gitbook/assets/aws marketplace.png" alt=""><figcaption></figcaption></figure> |
| 23 | + |
| 24 | +### PreRequisite |
| 25 | + |
| 26 | +* A private Amazon Redshift Database (Serverless or Provisioned) is created and accessible from the VPC and Subnet where the Ockam Node will be launched. |
| 27 | +* Security Group associated with the Amazon Redshift Database allows inbound traffic on the required default port (5439) from the subnet where the Ockam Outlet Node will reside. |
| 28 | +* You have permission to subscribe and launch Cloudformation stack from AWS Marketplace on the AWS Account running Amazon Redshift. |
| 29 | + |
| 30 | +### Create an Orchestrator Project |
| 31 | + |
| 32 | +1. [Sign up for Ockam](https://www.ockam.io/download) and pick a subscription plan through the guided workflow on Ockam.io. |
| 33 | +2. Run the following commands to install Ockam Command and enroll with the Ockam Orchestrator. |
| 34 | + |
| 35 | +``` |
| 36 | +curl --proto '=https' --tlsv1.2 -sSfL https://install.command.ockam.io | bash |
| 37 | +source "$HOME/.ockam/env" |
| 38 | +
|
| 39 | +ockam enroll |
| 40 | +``` |
| 41 | + |
| 42 | +3. Control which identities are allowed to enroll themselves into your project by issuing unique one-time use enrollment tickets. Generate two enrollment tickets, one for the Outlet and one for the Inlet. |
| 43 | + |
| 44 | +``` |
| 45 | +# Enrollment ticket for Ockam Outlet Node |
| 46 | +ockam project ticket --expires-in 10h --usage-count 1 \ |
| 47 | + --attribute amazon-redshift-outlet \ |
| 48 | + --relay redshift \ |
| 49 | + > "outlet.ticket" |
| 50 | +
|
| 51 | +# Enrollment ticket for Ockam Inlet Node |
| 52 | +ockam project ticket --expires-in 10h --usage-count 1 \ |
| 53 | + --attribute amazon-redshift-inlet \ |
| 54 | + > "inlet.ticket" |
| 55 | +``` |
| 56 | + |
| 57 | +### Setup Ockam Redshift Outlet Node |
| 58 | + |
| 59 | +* Login to AWS Account you would like to use |
| 60 | +* Subscribe to "Ockam - Node for Amazon Redshift" in AWS Marketplace  |
| 61 | +* Navigate to `AWS Marketplace -> Manage subscriptions`. Select `Ockam - Node for Amazon Redshift` from the list of subscriptions. Select `Actions-> Launch Cloudformation stack`  |
| 62 | +* Select the Region you want to deploy and click `Continue to Launch`. Under Actions, select `Launch Cloudformation` |
| 63 | +* Create stack with the following details |
| 64 | + * **Stack name**: `redshift-ockam-outlet` or any name you prefer |
| 65 | + * Network Configuration |
| 66 | + * **VPC ID:** Choose a VPC ID where the EC2 instance will be deployed. |
| 67 | + * **Subnet ID:** Select a suitable Subnet ID within the chosen VPC that has access to Amazon Redshift. _Note: Security Group associated with Amazon Redshift should allow inbound traffic on the required default port (5439) from the IP address of the Subnet or VPC_. |
| 68 | + * **EC2 Instance Type**: Default instance type is `m6a.large`. If you would like predictable network bandwidth of 12.5 Gbps please use `m6a.8xlarge` or a small instance type like `t3.medium` depending on your use case |
| 69 | + * Ockam Node Configuration |
| 70 | + * **Enrollment ticket**: Copy and paste the content of the `outlet.ticket` generated above |
| 71 | + * **Redshift Database Endpoint:** To configure the Ockam Redshift Outlet Node, you'll need to specify the Amazon Redshift Endpoint. This configuration allows the Ockam Redshift Outlet Node to connect to the database.  |
| 72 | + * Example: `cluster-name.xxxx.region.redshift.amazonaws.com:5439` or `workgroup.account.region.redshift-serverless.amazonaws.com:5439` |
| 73 | + * _Note: If you are copy pasting the Redshift Endpoint value from the AWS Console, please make sure to remove the /DATABASE\_NAME at the end as it is not needed_ |
| 74 | + * **JSON Node Configuration**: Copy and paste the below configuration. Note that the configuration values match with the enrollment tickets created in the previous step. `$REDSHIFT_ENDPOINT` will be replaced during runtime. |
| 75 | + |
| 76 | +```json |
| 77 | +{ |
| 78 | + "http-server-port": 23345, |
| 79 | + "relay": "redshift", |
| 80 | + "tcp-outlet": { |
| 81 | + "to": "$REDSHIFT_ENDPOINT", |
| 82 | + "allow": "amazon-redshift-inlet" |
| 83 | + } |
| 84 | +} |
| 85 | +``` |
| 86 | + |
| 87 | +* Click Next to launch the CloudFormation run. |
| 88 | +* A successful CloudFormation stack run configures the Ockam Redshift Outlet node on an EC2 machine. |
| 89 | + * EC2 machine mounts an EFS volume created in the same subnet. Ockam state is stored in the EFS volume. |
| 90 | + * A security group with egress access to the internet will be attached to the EC2 machine. |
| 91 | +* Connect to the EC2 machine via AWS Session Manager.  |
| 92 | + * To view the log file, run `sudo cat /var/log/cloud-init-output.log`. |
| 93 | + * _Note: DNS Resolution for the EFS drive may take upto 10 minutes, you will see the script retrying every 30 seconds to resolve_  |
| 94 | + * Successful run will show `Ockam node setup completed successfully` in the logs |
| 95 | + * To view the status of Ockam node run `curl http://localhost:23345/show | jq` |
| 96 | + * View the Ockam node status in CloudWatch. |
| 97 | + * Navigate to `Cloudwatch -> Log Group` and select `redshift-ockam-outlet-status-logs`. Select the Logstream for the EC2 instance.  |
| 98 | + * The Cloudformation template creates a subscription filter that sends data to a Cloudwatch alarm `redshift-ockam-outlet-OckamNodeDownAlarm.`Alarm will turn green upon ockam node successfully running.  |
| 99 | + * An Autoscaling group ensures atleast one EC2 instance is running at all times. |
| 100 | + |
| 101 | +Ockam redshift outlet node setup is complete. You can now create Ockam redshisft inlet nodes in any network to establish secure communication. |
| 102 | + |
| 103 | +### Setup Ockam Inlet Node  |
| 104 | + |
| 105 | +You can set up an Ockam Redshift Inlet Node either in AWS or locally using Docker. Here are both options: |
| 106 | + |
| 107 | +**Option 1: Setup Inlet Node in AWS** |
| 108 | + |
| 109 | +* Login to AWS Account you would like to use |
| 110 | +* Subscribe to "[Ockam - Node" ](https://aws.amazon.com/marketplace/pp/prodview-gov6p3wh224ho?sr=0-1\&ref\_=beagle\&applicationId=AWSMPContessa) in AWS Marketplace  |
| 111 | +* Navigate to `AWS Marketplace -> Manage subscriptions`. Select `Ockam - Node` from the list of subscriptions. Select `Actions-> Launch Cloudformation stack`  |
| 112 | +* Select the Region you want to deploy and click `Continue to Launch`. Under Actions, select `Launch Cloudformation` |
| 113 | +* Create stack with below details |
| 114 | + * Stack name: `redshift-ockam-inlet` or any name you prefer |
| 115 | + * Network Configuration |
| 116 | + * Select suitable values for `VPC ID` and `Subnet ID` |
| 117 | + * **EC2 Instance Type**: Default instance type is `m6a.large`. If you would like predictable network bandwidth of 12.5 Gbps please use `m6a.8xlarge` or a small instance type like `t3.medium` depending on your use case |
| 118 | + * Ockam Configuration |
| 119 | + * **Enrollment ticket**: Copy and paste the content of the `inlet.ticket` generated above |
| 120 | + * **JSON Node Configuration**: Copy and paste the below configuration.   |
| 121 | + |
| 122 | +``` |
| 123 | +{ |
| 124 | + "http-server-port": 23345, |
| 125 | + "tcp-inlet": { |
| 126 | + "from": "0.0.0.0:15439", |
| 127 | + "via": "redshift", |
| 128 | + "allow": "amazon-redshift-outlet" |
| 129 | + } |
| 130 | + } |
| 131 | +``` |
| 132 | + |
| 133 | +* Click Next to launch the CloudFormation run. |
| 134 | +* A successful CloudFormation stack run configures the Ockam inlet node on an EC2 machine. |
| 135 | +* EC2 machine mounts an EFS volume created in the same subnet. Ockam state is stored in the EFS volume. |
| 136 | +* Connect to the EC2 machine via AWS Session Manager.  |
| 137 | + * To view the log file, run `sudo cat /var/log/cloud-init-output.log`. |
| 138 | + * Successful run will show `Ockam node setup completed successfully` in the logs |
| 139 | + * To view the status of Ockam node run `curl http://localhost:23345/show | jq` |
| 140 | +* View the Ockam node status in CloudWatch. |
| 141 | + * Navigate to `Cloudwatch -> Log Group` and select `redshift-ockam-inlet-status-logs`. Select the Logstream for the EC2 instance.  |
| 142 | + * Cloudformation template creates a subscription filter that sends data to a Cloudwatch alarm `redshift-ockam-inlet-OckamNodeDownAlarm.`Alarm will turn green upon ockam node successfully running.  |
| 143 | +* An Autoscaling group ensures atleast one EC2 instance is running at all times.  |
| 144 | + |
| 145 | +Use any `postgresql`client and connect to `localhost:15432 (`PGHOST=`localhost,` PGPORT=15439) from the machine running the Ockam redshift Inlet node. |
| 146 | + |
| 147 | +**Option 2: Setup Inlet Node Locally with Docker Compose** |
| 148 | + |
| 149 | +To set up an Inlet Node locally and interact with it outside of AWS, use Docker Compose.  |
| 150 | + |
| 151 | +* Create a file named `docker-compose.yml` with the following content: |
| 152 | + |
| 153 | +```yaml |
| 154 | +services: |
| 155 | + ockam: |
| 156 | + image: ghcr.io/build-trust/ockam |
| 157 | + container_name: redshift-inlet |
| 158 | + environment: |
| 159 | + ENROLLMENT_TICKET: ${ENROLLMENT_TICKET:-} |
| 160 | + OCKAM_LOGGING: true |
| 161 | + OCKAM_LOG_LEVEL: info |
| 162 | + command: |
| 163 | + - node |
| 164 | + - create |
| 165 | + - --enrollment-ticket |
| 166 | + - ${ENROLLMENT_TICKET} |
| 167 | + - --foreground |
| 168 | + - --configuration |
| 169 | + - | |
| 170 | + tcp-inlet: |
| 171 | + via: redshift |
| 172 | + allow: amazon-redshift-outlet |
| 173 | + from: 127.0.0.1:15439 |
| 174 | + network_mode: host |
| 175 | + |
| 176 | + psql-client: |
| 177 | + image: postgres |
| 178 | + container_name: psql-client |
| 179 | + command: /bin/bash -c "while true; do sleep 30; done" |
| 180 | + depends_on: |
| 181 | + - ockam |
| 182 | + network_mode: host |
| 183 | +``` |
| 184 | +
|
| 185 | +* Run the following command from the same location as the `docker-compose.yml` and the `inlet.ticket` to create an Ockam postgres inlet that can connect to the outlet running in AWS , along with psql client container.  |
| 186 | + |
| 187 | +```bash |
| 188 | +ENROLLMENT_TICKET=$(cat inlet.ticket) docker-compose up -d |
| 189 | +``` |
| 190 | + |
| 191 | +* Check status of Ockam inlet node. You will see `The node is UP` when ockam is configured successfully and ready to accept connection |
| 192 | + |
| 193 | +``` |
| 194 | +docker exec -it redshift-inlet /ockam node show |
| 195 | +``` |
| 196 | +
|
| 197 | +* Connect to **psql-client** container and run commands |
| 198 | +
|
| 199 | +```bash |
| 200 | +# Connect to the container |
| 201 | +docker exec -it psql-client /bin/bash |
| 202 | +
|
| 203 | +# Update the *_REPLACE placeholder variables |
| 204 | +export PGUSER="PGUSER_REPLACE"; |
| 205 | +export PGPASSWORD="PGPASSWORD_REPLACE"; |
| 206 | +export PGDATABASE="PGDATABASE_REPLACE"; |
| 207 | +export PGHOST="localhost"; |
| 208 | +export PGPORT="15439"; |
| 209 | +
|
| 210 | +# list tables |
| 211 | +psql -c "\dt"; |
| 212 | +
|
| 213 | +# Create a table |
| 214 | +psql -c "CREATE TABLE __test__ (key VARCHAR(255), value VARCHAR(255));"; |
| 215 | +
|
| 216 | +# Insert some data |
| 217 | +psql -c "INSERT INTO __test__ (key, value) VALUES ('0', 'Hello');"; |
| 218 | +
|
| 219 | +# Query the data |
| 220 | +psql -c "SELECT * FROM __test__;"; |
| 221 | +
|
| 222 | +# Drop table if it exists |
| 223 | +psql -c "DROP TABLE IF EXISTS __test__;"; |
| 224 | +``` |
| 225 | + |
| 226 | +This setup allows you to run an Ockam Redshift Inlet Node locally and communicate securely with a private Amazon Redshift database running in AWS |
| 227 | + |
| 228 | +* Cleanup  |
| 229 | + |
| 230 | +``` |
| 231 | +docker compose down --volumes --remove-orphans |
| 232 | +``` |
| 233 | + |
0 commit comments