|
1 | 1 | # Confluent Cloud for Apache Flink (CCAF) Flight Consolidator App Lambda |
| 2 | +This repository empowers the deployment of a robust [Flink Job Graph](https://github.com/j3-signalroom/j3-techstack-lexicon/blob/main/apache-flink-glossary.md#jobgraph) to [Confluent Cloud for Apache Flink (CCAF)](https://docs.confluent.io/cloud/current/flink/overview.html) , enabling seamless and continuous real-time streaming. The [Flink Job Graph](https://github.com/j3-signalroom/j3-techstack-lexicon/blob/main/apache-flink-glossary.md#jobgraph) meticulously ingests flight data from the `airline.sunset_avro` and `airline.skyone_avro` Kafka topics, standardizing and unifying the information into a single, consolidated `airline.flight_avro` Kafka topic. By leveraging advanced stream processing capabilities, this deployment ensures high scalability, data consistency, and reliability, providing organizations with a powerful foundation for actionable flight analytics and insightful decision-making. |
| 3 | + |
| 4 | +--- |
| 5 | + |
| 6 | +**Table of Contents** |
| 7 | + |
| 8 | +<!-- toc --> |
| 9 | ++ [1.0 Let's get started!](#10-lets-get-started) |
| 10 | ++ [2.0 Visualizing the Terraform Configuration](#20-visualizing-the-terraform-configuration) |
| 11 | +<!-- tocstop --> |
| 12 | + |
| 13 | +## 1.0 Let's get started! |
| 14 | +1. Take care of the cloud and local environment prequisities listed below: |
| 15 | + > You need to have the following cloud accounts: |
| 16 | + > - [AWS Account](https://signin.aws.amazon.com/) *with SSO configured* |
| 17 | + > [`aws2-wrap` utility](https://pypi.org/project/aws2-wrap/#description) |
| 18 | +
|
| 19 | + > You need to have the following installed on your local machine: |
| 20 | + > - [AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) |
| 21 | +
|
| 22 | +2. Clone the repo: |
| 23 | + ```bash |
| 24 | + git clone https://github.com/j3-signalroom/ccaf_kickstarter-flight_consolidator_app-lambda.git |
| 25 | + ``` |
| 26 | + |
| 27 | +3. **Navigate to the Root Directory**: Open your Terminal and navigate to the root folder of the `ccaf_kickstarter-flight_consolidator_app-lambda/` repository that you have cloned. You can do this by executing: |
| 28 | + |
| 29 | + ```bash |
| 30 | + cd path/to/ccaf_kickstarter-flight_consolidator_app-lambda/ |
| 31 | + ``` |
| 32 | + |
| 33 | + Replace `path/to/` with the actual path where your repository is located. |
| 34 | + |
| 35 | +4. **Run the Script to Publish (Create) or Unpublish (Delete) the ECR Repository:** Execute the `build-publish-lambda.sh` script to create an AWS Elastic Container Registry (ECR) repository, build the AWS Lambda Docker container, and publish it to the newly created ECR repository. This will make the container image available for future deployments. |
| 36 | + |
| 37 | + Use the following command format: |
| 38 | + |
| 39 | + ```bash |
| 40 | + scripts/build-publish-lambda.sh <create | delete> --profile=<SSO_PROFILE_NAME> |
| 41 | + ``` |
| 42 | + |
| 43 | +5. **Replace Argument Placeholders:** |
| 44 | + - `<create | delete>`: Specify either `create` to create the ECR repository or `delete` to remove it. |
| 45 | + - `<SSO_PROFILE_NAME>`: Replace this with your AWS Single Sign-On (SSO) profile name, which identifies your hosted AWS infrastructure. |
| 46 | + |
| 47 | + For example, to create the ECR repository, use the following command: |
| 48 | + ```bash |
| 49 | + scripts/build-publish-lambda.sh create --profile=my-aws-sso-profile |
| 50 | + ``` |
| 51 | + Replace `my-aws-sso-profile` with your actual AWS SSO profile name. |
| 52 | + |
| 53 | +6. **Run the Script to Create or Destroy the Terraform Configuration:** Execute the `run-terraform-locally.sh` script to create the IAM Policy and Role for the Lambda, and then invoke (run) the Lambda. |
| 54 | + |
| 55 | + Use the following command format: |
| 56 | + |
| 57 | + ```bash |
| 58 | + scripts/run-terraform-locally.sh <create | delete> --profile=<SSO_PROFILE_NAME> --catalog-name=<CATALOG_NAME> --database-name=<DATABASE_NAME> --ccaf-secrets-path=<CCAF_SECRETS_PATH>" |
| 59 | + ``` |
| 60 | +
|
| 61 | +7. **Replace Argument Placeholders:** |
| 62 | + - `<create | delete>`: Specify either `create` to create the ECR repository or `delete` to remove it. |
| 63 | + - `<SSO_PROFILE_NAME>`: Replace this with your AWS Single Sign-On (SSO) profile name, which identifies your hosted AWS infrastructure. |
| 64 | + - `<CATALOG_NAME>`: Replace this with the name of your Flink catalog. |
| 65 | + - `<DATABASE_NAME>`: Replace this with the name of your Flink database. |
| 66 | + - `<CCAF_SECRETS_PATH>`: Replace this with the path to the Confluent Cloud for Apache Flink (CCAF) AWS Secrets Manager secrets. |
| 67 | +
|
| 68 | + For example, to create the IAM Policy and Role for the Lambda, and then invoke (run) the Lambda, use the following command: |
| 69 | + ```bash |
| 70 | + scripts/run-terraform-locally.sh create --profile=my-aws-sso-profile --catalog-name=flink_kickstarter --database-name=flink_kickstarter --ccaf-secrets-path="/confluent_cloud_resource/flink_kickstarter/flink_compute_pool" |
| 71 | + ``` |
| 72 | + Replace `my-aws-sso-profile` with your actual AWS SSO profile name, `flink_kickstart` Flink catalog (Environment), `flink_kickstart` Flink database (Kafka Cluster) and the `/confluent_cloud_resource/flink_kickstarter/flink_compute_pool` AWS Secrets Manager secrets path. |
| 73 | +
|
| 74 | +8. Or, to run the similiar script from GitHub, follow these steps: |
| 75 | +
|
| 76 | + a. Deploy the Repository: Ensure that you have cloned or forked the repository to your GitHub account. |
| 77 | +
|
| 78 | + b. Set Required Secrets and Variables: Before running any of the GitHub workflows provided in the repository, you must define at least the `AWS_DEV_ACCOUNT_ID` variable (which should contain your AWS Account ID for your development environment). To do this: |
| 79 | +
|
| 80 | + - Go to the Settings of your cloned or forked repository in GitHub. |
| 81 | +
|
| 82 | + - Navigate to **Secrets and Variables > Actions**. |
| 83 | +
|
| 84 | + - Add the `AWS_DEV_ACCOUNT_ID` and any other required variables or secrets. |
| 85 | +
|
| 86 | + c. Navigate to the **Actions Page**: |
| 87 | +
|
| 88 | + - From the cloned or forked repository on GitHub, click on the **Actions tab**. |
| 89 | +
|
| 90 | + d. **Select and Run the Deploy Workflow**: |
| 91 | +
|
| 92 | + - Find the **Deploy** workflow link on the left side of the **Actions** page and click on it. |
| 93 | +
|
| 94 | +  |
| 95 | +
|
| 96 | + - On the **Deploy** workflow page, click the **Run workflow** button. |
| 97 | +
|
| 98 | + - A workflow dialog box will appear. Fill in the necessary details and click **Run workflow** to initiate the building and publishing the Lambda docker container to ECR, to create the IAM Policy and Role for the Lambda, and then invoke (run) the Lambda. |
| 99 | +
|
| 100 | +  |
| 101 | +
|
| 102 | +By following these steps, you will set up the necessary infrastructure to build and deploy the **Flight Consolidator Confluent Cloud for Apache Flink App**! |
| 103 | +
|
| 104 | +## 2.0 Visualizing the Terraform Configuration |
| 105 | +Below is the Terraform visualization of the Terraform configuration. It shows the resources and their dependencies, making the infrastructure setup easier to understand. |
| 106 | +
|
| 107 | + |
| 108 | +
|
| 109 | +> **To fully view the image, open it in another tab on your browser to zoom in.** |
| 110 | +
|
| 111 | +When you update the Terraform Configuration, to update the Terraform visualization, use the [`terraform graph`](https://developer.hashicorp.com/terraform/cli/commands/graph) command with [Graphviz](https://graphviz.org/) to generate a visual representation of the resources and their dependencies. To do this, run the following command: |
| 112 | +
|
| 113 | +```bash |
| 114 | +terraform graph | dot -Tpng > .blog/images/terraform-visualization.png |
| 115 | +``` |
0 commit comments