Skip to content

Commit 16667a5

Browse files
authored
DOC-1186 Redpanda Connect for BYOVPC on AWS (#274)
* Add config to AWS BYOVPC docs for RPCN * Correct IAM instance profile list * PM review comments * PM review comments * minor rewording * PM review comments * Update note * docs review comments
1 parent e176361 commit 16667a5

File tree

5 files changed

+125
-65
lines changed

5 files changed

+125
-65
lines changed

modules/develop/partials/availability-message.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
====
33
44
* Redpanda Connect is available in limited availability (LA) for BYOC and Dedicated clusters. Features in LA are production-ready and are covered by Redpanda Support for early adopters. To unlock Redpanda Connect for your account, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^].
5-
* Redpanda Connect is in beta for BYOVPC clusters on GCP but not on AWS or Azure.
5+
* Redpanda Connect is in beta for BYOVPC clusters on AWS and GCP. It is not yet available for BYOVPC clusters on Azure.
66
* Redpanda Connect is available in beta for Serverless clusters. Features in beta are not covered by Redpanda Support and should not be used in production environments.
77
====

modules/get-started/pages/cloud-overview.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -264,7 +264,7 @@ Features in limited availability are production-ready and are covered by Redpand
264264

265265
The following features are currently in limited availability in Redpanda Cloud:
266266

267-
* Redpanda Connect for Dedicated and BYOC (not BYOVPC on AWS or Azure)
267+
* Redpanda Connect for Dedicated and BYOC (excluding BYOVPC on Azure)
268268
* Serverless
269269
* Dedicated and BYOC for Azure
270270
* BYOVPC for GCP
@@ -276,7 +276,7 @@ Features in beta are available for testing and feedback. They are not covered by
276276

277277
The following features are currently in beta in Redpanda Cloud:
278278

279-
* Redpanda Connect for BYOVPC on GCP and Serverless
279+
* Redpanda Connect on Serverless, and for BYOVPC on GCP and AWS
280280
* Redpanda Terraform provider
281281
* BYOVPC for AWS and Azure
282282
* Integration with Apache Iceberg

modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc

Lines changed: 117 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,21 @@ The https://github.com/redpanda-data/cloud-examples/blob/main/customer-managed/a
5252

5353
NOTE: For simplicity, these instructions assume that Terraform is configured to use local state. You may want to configure https://developer.hashicorp.com/terraform/language/state/remote[remote state^].
5454

55-
Define a JSON file called `byovnet.auto.tfvars.json` inside the Terraform directory that contains information about the VPC. Optionally, you can enable PrivateLink. For example:
55+
Define a JSON file called `byovnet.auto.tfvars.json` inside the Terraform directory that contains information about the VPC.
56+
57+
You can update the example configuration to customize your setup:
58+
59+
- Enable PrivateLink (`"enable_private_link": true`).
60+
- Preserve cluster data when deleting the cluster (`"force_destroy_cloud_storage": false`).
61+
- Use https://docs.aws.amazon.com/IAM/latest/UserGuide/access_iam-tags.html[condition tags^] to control resource modifications based on AWS tags. For example:
62+
+
63+
```json
64+
"condition_tags": {
65+
"redpanda-managed": "true"
66+
},
67+
```
68+
69+
Example configuration:
5670

5771
```json
5872
cat > byoc.auto.tfvars.json <<EOF
@@ -67,15 +81,32 @@ cat > byoc.auto.tfvars.json <<EOF
6781
"ignore_tags": [
6882
],
6983
"vpc_id": "${AWS_VPC_ID}",
70-
"zones": [],
71-
"enable_private_link": true,
84+
"zones": [
85+
"use2-az1",
86+
"use2-az2",
87+
"use2-az3"
88+
],
89+
"create_rpk_user": true,
90+
"enable_redpanda_connect": true,
91+
"enable_private_link": false,
7292
"create_rpk_user": true,
73-
"force_destroy_cloud_storage": true
93+
"force_destroy_cloud_storage": true,
94+
"public_subnet_cidrs": [
95+
"10.0.1.0/24",
96+
"10.0.3.0/24",
97+
"10.0.5.0/24",
98+
"10.0.7.0/24",
99+
"10.0.9.0/24",
100+
"10.0.11.0/24"
101+
]
74102
}
75103
EOF
76104

77105
```
78106

107+
NOTE: At least one public subnet is required to create a cluster. The example configuration includes multiple public subnets to allow for future scaling. In addition, the VPC should have an Internet Gateway and an associated Route Table that exposes traffic into the VPC, and allows the Redpanda Control Plane to access the cluster.
108+
109+
79110
== Deploy Terraform
80111

81112
Initialize, plan, and apply Terraform to set up the AWS infrastructure:
@@ -130,6 +161,7 @@ To create the Redpanda network:
130161
```json
131162
cat > redpanda-network.json <<EOF
132163
{
164+
"network": {
133165
"name":"sample-redpanda-network",
134166
"resource_group_id": "${REDPANDA_RG_ID}",
135167
"cloud_provider":"CLOUD_PROVIDER_AWS",
@@ -144,13 +176,14 @@ cat > redpanda-network.json <<EOF
144176
"arn": "${AWS_DYNAMODB_TABLE}"
145177
},
146178
"private_subnets": {
147-
"arns": "${AWS_PRIVATE_SUBNETS}"
179+
"arns": ${AWS_PRIVATE_SUBNETS}
148180
},
149181
"vpc": {
150182
"arn": "${AWS_VPC}"
151183
}
152184
}
153-
}
185+
}
186+
}
154187
}
155188
EOF
156189
```
@@ -162,7 +195,7 @@ export REDPANDA_NETWORK_ID=$(curl -X POST "https://api.redpanda.com/v1/networks"
162195
-H "accept: application/json" \
163196
-H "content-type: application/json" \
164197
-H "authorization: Bearer ${BEARER_TOKEN}" \
165-
--data-binary @redpanda-network.json | jq -r '.operation.id')
198+
--data-binary @redpanda-network.json | jq -r '.operation.resource_id')
166199
```
167200
+
168201
The Create Network request returns a `resource_id`. For example:
@@ -188,13 +221,28 @@ The Create Network request returns a `resource_id`. For example:
188221

189222
To create the Redpanda cluster:
190223

191-
. Create environment variables for cluster information, like the version, tier, and availability zones.
224+
. For all cluster information, including the version, tier, storage, security, and availability zones, create the following environment variables with values from Terraform. You can customize the cluster name, Repdanda version, usage tier, and AWS zones for your cluster using the environment variables.
192225
+
193226
```bash
194-
export AWS_ZONES='["use-az1", "use-az2", "use-az3"]'
227+
export AWS_ZONES='["use2-az1", "use2-az2", "use2-az3"]'
195228
export REDPANDA_CLUSTER_NAME=sample-redpanda-cluster
196-
export REDPANDA_VERSION=24.3
229+
export REDPANDA_VERSION=25.1
197230
export REDPANDA_THROUGHPUT_TIER=tier-1-aws-v3-arm
231+
export AGENT_INSTANCE_PROFILE_ARN="$(terraform output -raw agent_instance_profile_arn)"
232+
export CONNECTORS_NODE_GROUP_INSTANCE_PROFILE="$(terraform output -raw connectors_node_group_instance_profile_arn)"
233+
export REDPANDA_NODE_GROUP_INSTANCE_PROFILE="$(terraform output -raw redpanda_node_group_instance_profile_arn)"
234+
export REDPANDA_CONNECT_NODE_GROUP_INSTANCE_PROFILE="$(terraform output -raw redpanda_connect_node_group_instance_profile_arn)"
235+
export UTILITY_NODE_GROUP_INSTANCE_PROFILE="$(terraform output -raw utility_node_group_instance_profile_arn)"
236+
export CONNECTORS_SECURITY_GROUP="$(terraform output -raw connectors_security_group_arn)"
237+
export REDPANDA_CONNECT_SECURITY_GROUP="$(terraform output -raw redpanda_connect_security_group_arn)"
238+
export NODE_SECURITY_GROUP="$(terraform output -raw node_security_group_arn)"
239+
export UTILITY_SECURITY_GROUP="$(terraform output -raw utility_security_group_arn)"
240+
export REDPANDA_AGENT_SECURITY_GROUP="$(terraform output -raw redpanda_agent_security_group_arn)"
241+
export REDPANDA_NODE_GROUP_SECURITY_GROUP="$(terraform output -raw redpanda_node_group_security_group_arn)"
242+
export CLUSTER_SECURITY_GROUP="$(terraform output -raw cluster_security_group_arn)"
243+
export K8S_CLUSTER_ROLE="$(terraform output -raw k8s_cluster_role_arn)"
244+
export CLOUD_STORAGE_BUCKET="$(terraform output -raw cloud_storage_bucket_arn)"
245+
export PERMISSIONS_BOUNDARY_POLICY="$(terraform output -raw permissions_boundary_policy_arn)"
198246
```
199247
+
200248
TIP: See the full list of zones and tiers available with each provider in the xref:api:ROOT:cloud-controlplane-api.adoc#api-description[Control Plane API reference].
@@ -204,56 +252,64 @@ TIP: See the full list of zones and tiers available with each provider in the xr
204252
```json
205253
cat > redpanda-cluster.json <<EOF
206254
{
207-
"cloud_provider":"CLOUD_PROVIDER_AWS",
208-
"connection_type":"CONNECTION_TYPE_PRIVATE",
209-
"name": "${REDPANDA_CLUSTER_NAME}",
210-
"resource_group_id": "${REDPANDA_RG_ID}",
211-
"network_id": "${REDPANDA_NETWORK_ID}",
212-
"region": "${AWS_REGION}",
213-
"throughput_tier": "${REDPANDA_THROUGHPUT_TIER}",
214-
"type": "TYPE_BYOC",
215-
"zones": ${AWS_ZONES},
216-
"redpanda_version": "${REDPANDA_VERSION}",
217-
"customer_managed_resources": {
218-
"aws": {
219-
"agent_instance_profile": {
220-
"arn": "<agent_instance_profile_arn from terraform outputs>"
221-
},
222-
"connectors_node_group_instance_profile": {
223-
"arn": "<connectors_node_group_instance_profile_arn from terraform outputs>"
224-
},
225-
"redpanda_node_group_instance_profile": {
226-
"arn": "<redpanda_node_group_instance_profile_arn from terraform outputs>"
227-
},
228-
"utility_node_group_instance_profile": {
229-
"arn": "<utility_node_group_instance_profile_arn from terraform outputs>"
230-
},
231-
"connectors_security_group": {
232-
"arn": "<connectors_security_group_arn from terraform outputs>"
233-
},
234-
"node_security_group": {
235-
"arn": "<node_security_group_arn from terraform outputs>"
236-
},
237-
"utility_security_group": {
238-
"arn": "<utility_security_group_arn from terraform outputs>"
239-
},
240-
"redpanda_agent_security_group": {
241-
"arn": "<redpanda_agent_security_group_arn from terraform outputs>"
242-
},
243-
"redpanda_node_group_security_group": {
244-
"arn": "<redpanda_node_group_security_group_arn from terraform outputs>"
245-
},
246-
"cluster_security_group": {
247-
"arn": "<cluster_security_group_arn from terraform outputs>"
248-
},
249-
"k8s_cluster_role": {
250-
"arn": "<k8s_cluster_role_arn from terraform outputs>"
251-
},
252-
"cloud_storage_bucket": {
253-
"arn": "<cloud_storage_bucket_arn from terraform outputs>"
254-
},
255-
"permissions_boundary_policy": {
256-
"arn": "<permissions_boundary_policy_arn from terraform outputs>"
255+
"cluster": {
256+
"cloud_provider":"CLOUD_PROVIDER_AWS",
257+
"connection_type":"CONNECTION_TYPE_PRIVATE",
258+
"name": "${REDPANDA_CLUSTER_NAME}",
259+
"resource_group_id": "${REDPANDA_RG_ID}",
260+
"network_id": "${REDPANDA_NETWORK_ID}",
261+
"region": "${AWS_REGION}",
262+
"throughput_tier": "${REDPANDA_THROUGHPUT_TIER}",
263+
"type": "TYPE_BYOC",
264+
"zones": ${AWS_ZONES},
265+
"redpanda_version": "${REDPANDA_VERSION}",
266+
"customer_managed_resources": {
267+
"aws": {
268+
"agent_instance_profile": {
269+
"arn": "${AGENT_INSTANCE_PROFILE_ARN}"
270+
},
271+
"connectors_node_group_instance_profile": {
272+
"arn": "${CONNECTORS_NODE_GROUP_INSTANCE_PROFILE}"
273+
},
274+
"redpanda_connect_node_group_instance_profile": {
275+
"arn": "${REDPANDA_CONNECT_NODE_GROUP_INSTANCE_PROFILE}"
276+
},
277+
"redpanda_node_group_instance_profile": {
278+
"arn": "${REDPANDA_NODE_GROUP_INSTANCE_PROFILE}"
279+
},
280+
"utility_node_group_instance_profile": {
281+
"arn": "${UTILITY_NODE_GROUP_INSTANCE_PROFILE}"
282+
},
283+
"connectors_security_group": {
284+
"arn": "${CONNECTORS_SECURITY_GROUP}"
285+
},
286+
"redpanda_connect_security_group": {
287+
"arn": "${REDPANDA_CONNECT_SECURITY_GROUP}"
288+
},
289+
"node_security_group": {
290+
"arn": "${NODE_SECURITY_GROUP}"
291+
},
292+
"utility_security_group": {
293+
"arn": "${UTILITY_SECURITY_GROUP}"
294+
},
295+
"redpanda_agent_security_group": {
296+
"arn": "${REDPANDA_AGENT_SECURITY_GROUP}"
297+
},
298+
"redpanda_node_group_security_group": {
299+
"arn": "${REDPANDA_NODE_GROUP_SECURITY_GROUP}"
300+
},
301+
"cluster_security_group": {
302+
"arn": "${CLUSTER_SECURITY_GROUP}"
303+
},
304+
"k8s_cluster_role": {
305+
"arn": "${K8S_CLUSTER_ROLE}"
306+
},
307+
"cloud_storage_bucket": {
308+
"arn": "${CLOUD_STORAGE_BUCKET}"
309+
},
310+
"permissions_boundary_policy": {
311+
"arn": "${PERMISSIONS_BOUNDARY_POLICY}"
312+
}
257313
}
258314
}
259315
}
@@ -342,7 +398,7 @@ The `rpk cloud byoc aws apply` command performs validation checks before proceed
342398
* RPK user: Checks if the user running the command has sufficient privileges to provision the agent. Any
343399
missing permissions are displayed in the output.
344400

345-
* IAM instance profile: Checks that `connectors_node_group_instance_profile`, `redpanda_node_group_instance_profile`,
401+
* IAM instance profile: Checks that `connectors_node_group_instance_profile`, `redpanda_node_group_instance_profile`, `redpanda_connect_node_group_instance_profile`,
346402
`utility_node_group_instance_profile`, and `k8s_cluster_role` have the minimum required permissions. Any missing permissions are displayed in the output.
347403

348404
* Storage: Checks that the `management_bucket` exists and is versioned, checks that the `cloud_storage_bucket` exists and is not versioned, and checks that the `dynamodb_table` exists.

modules/get-started/pages/whats-new-cloud.adoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ Data transforms are supported for BYOC and Dedicated clusters running Redpanda v
4141

4242
Redpanda Cloud is starting to introduce beta versions of xref:develop:agents/about.adoc[AI agents] for enterprise agentic applications driven by a continuous data feed.
4343

44+
=== Redpanda Connect for BYOVPC on AWS and GCP: beta
45+
46+
Redpanda Connect is now enabled when you create a BYOVPC cluster on xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc[AWS] or xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[GCP]. You can also add Redpanda Connect to an xref:get-started:cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc[existing BYOVPC GCP cluster].
47+
4448
== March 2025
4549

4650
=== Serverless

modules/shared/partials/feature-flag-rpcn.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@
22
====
33
44
* BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or https://www.redpanda.com/price-estimator[Redpanda Sales^].
5-
* Redpanda Connect is in beta for BYOVPC clusters on GCP.
5+
* Redpanda Connect is in beta for BYOVPC clusters on AWS and GCP. It is not yet available for BYOVPC clusters on Azure.
66
====

0 commit comments

Comments
 (0)