Skip to content

Commit c65341d

Browse files
micheleRPFeediver1frenchfrywpepperDeflaimun
authored
BYOVPC for AWS (#12)
* 550 byovpc for AWS * update cloud overview * Update modules/get-started/pages/cluster-types/byoc/create-byoc-cluster-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/whats-new-cloud.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * incorporate suggestions from doc review * BYOVPC for AWS (#47) * Insert newline before Prerequisites Header is not rendering due to missing newline. * Replace rpk user code with link to cloud-examples * Replace Configure your VPC section Modify this section to be the place where we refer the user to the cloud-examples repo for the terraform code. In the future this section can contain 2 tabs: one for the cloud-examples/terraform approach and one with aws cli commands. * Update the network/cluster create curl commands Some fields have been removed since this was written. Also fixed some syntax issues preventing the command vars from displaying properly. * Update the post-creation curl commands * Add information about rpk validation (#48) Adds the rpk validation output to the sample. Adds an explanation of what the validation will be checking. * doc edits * nest AWS, Azure, GCP folders under create BYOC also add aliases & update links to new locations * Update modules/get-started/pages/cluster-types/byoc/AWS/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/AWS/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/AWS/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * add glossterm * incorporate feedback from doc review * build with fixed glossterm * Updates to public API call 1. Remove redpanda_cloud_storage_manager_policy it is no longer used 2. Add "arn" prefix to connectors_node_group_instance_profile 3. Add permissions boundary field * incorporate feedback from Praseed + lowercase folders (in URL) * force folder rename * remove extra folders * re add vpc byo * add beta tag + minor text edits * edit What's New: BYOC VNet in LA --------- Co-authored-by: Joyce Fee <[email protected]> Co-authored-by: Sarah Haskins <[email protected]> Co-authored-by: Sarah Haskins <[email protected]> Co-authored-by: Paulo Borges <[email protected]>
1 parent f4cc281 commit c65341d

File tree

7 files changed

+274
-8
lines changed

7 files changed

+274
-8
lines changed

‎modules/ROOT/nav.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
** xref:get-started:cluster-types/byoc/index.adoc[BYOC]
1111
*** xref:get-started:cluster-types/byoc/aws/index.adoc[AWS]
1212
**** xref:get-started:cluster-types/byoc/aws/create-byoc-cluster-aws.adoc[]
13+
**** xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc[]
1314
*** xref:get-started:cluster-types/byoc/azure/index.adoc[Azure]
1415
**** xref:get-started:cluster-types/byoc/azure/create-byoc-cluster-azure.adoc[]
1516
**** xref:get-started:cluster-types/byoc/azure/vnet-azure.adoc[]

‎modules/get-started/pages/cloud-overview.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Redpanda offers three types of fully-managed cloud clusters:
1515
* <<Dedicated Cloud>>: Single-tenant clusters hosted in Redpanda Cloud. This provides more control over your deployment.
1616
* <<Bring Your Own Cloud (BYOC)>>: Clusters hosted in your private cloud. This provides more control over your deployment and offers full data sovereignty.
1717
+
18-
NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including service accounts, subnetworks, IAM roles, firewall rules, and storage buckets. For the most security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[customer-managed VPC].
18+
NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC or VNet, including subnetworks, IAM roles, and storage buckets/accounts. For the highest level of security, you can manage these resources yourself with a customer-managed VPC or VNet.
1919

2020
=== Serverless
2121

‎modules/get-started/pages/cluster-types/byoc/aws/create-byoc-cluster-aws.adoc‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,13 @@
22
:description: Use the Redpanda Cloud UI to create a BYOC cluster on AWS.
33
:page-aliases: get-started:cluster-types/byoc/create-byoc-cluster-aws.adoc, cloud:create-byoc-cluster-aws.adoc, deploy:deployment-option/cloud/create-byoc-cluster-aws.adoc
44

5-
To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`.
5+
To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: xref:get-started:cloud-overview.adoc#redpanda-cloud-architecture[Redpanda Cloud architecture].
66

7-
See also: xref:get-started:cloud-overview.adoc#redpanda-cloud-architecture[Redpanda Cloud architecture].
7+
NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a xref:./vpc-byo-aws.adoc[customer-managed VPC on AWS].
88

99
== Prerequisites
1010

11-
With BYOC, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. Before you deploy a BYOC cluster on AWS, check that the user creating the cluster has the following prerequisites:
11+
Before you deploy a BYOC cluster on AWS, check that the user creating the cluster has the following prerequisites:
1212

1313
* A minimum version of Redpanda `rpk` v24.1. See xref:manage:rpk/rpk-install.adoc[].
1414
* The permissions necessary to launch infrastructure for running Redpanda. See xref:security:authorization/cloud-iam-policies.adoc[IAM policies].

‎modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc‎

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
= Configure a Customer-Managed VPC on AWS
2+
:description: Connect Redpanda Cloud to your existing VPC for additional security.
3+
:page-aliases: get-started:cluster-types/byoc/vpc-byo-aws.adoc
4+
:page-cloud: true
5+
:page-beta: true
6+
7+
include::shared:partial$feature-flag.adoc[]
8+
9+
With a standard BYOC cluster, Redpanda manages the VPC lifecycle. For additional security, you can deploy the Redpanda glossterm:data plane[] into your existing shared VPC and manage the VPC lifecycle yourself. When you create a BYOC cluster, you specify your VPC and service account. The Redpanda Cloud agent doesn't create any new resources or alter any settings in your account. With a *customer-managed* VPC:
10+
11+
* You provide your own VPC in your AWS account.
12+
* You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters.
13+
* You control your security resources and policies, including subnets, service accounts, IAM roles, firewall rules, and storage buckets.
14+
15+
== Prerequisites
16+
17+
* Familiarity with the xref:api:ROOT:cloud-api.adoc[Redpanda Cloud API]. For example, you should familiarize yourself with how to use the Cloud API to authenticate and create a cluster.
18+
* Access to an AWS project in which you create your cluster.
19+
* Minimum permissions in that AWS project. For the actions required by the user who will create the cluster with `rpk cloud byoc aws apply`, see https://github.com/redpanda-data/cloud-examples/blob/main/customer-managed/aws/terraform/iam_rpk_user.tf[`iam_rpk_user.tf`^].
20+
21+
== Limitations
22+
23+
* You cannot move existing clusters to a customer-managed VPC.
24+
* You cannot change to a different VPC after creating a cluster with a customer-managed VPC.
25+
26+
== Create managed resources
27+
28+
For example Terraform code of the expected provisioned resources, see the https://github.com/redpanda-data/cloud-examples/tree/main/customer-managed/aws[customer-managed VPC on AWS README^].
29+
The ARNs (Amazon Resource Names) output by this Terraform code are necessary in later steps.
30+
31+
== Create a network with the Cloud API
32+
33+
After xref:deploy:deployment-option/cloud/api/cloud-api-authentication.adoc[authenticating] to the Cloud API, issue a create network request using the Terraform outputs.
34+
35+
For information about the fields, see
36+
https://redpanda-api-documentation.netlify.app/#post-/v1beta2/networks[Create Network].
37+
38+
```
39+
curl -X POST "https://api.redpanda.com/v1beta2/networks" \
40+
-H "accept: application/json"\
41+
-H "content-type: application/json" \
42+
-H "authorization: Bearer $YOUR_TOKEN" \
43+
-d @- << EOF
44+
{
45+
"name":"<your network resource name>",
46+
"resource_group_id": "<resource group id of the network>",
47+
"cloud_provider":"CLOUD_PROVIDER_AWS",
48+
"region":"<region>",
49+
"cluster_type":"TYPE_BYOC",
50+
"customer_managed_resources": {
51+
"aws": {
52+
"management_bucket": {
53+
"arn": "<management_bucket_arn from terraform outputs>"
54+
},
55+
"dynamodb_table": {
56+
"arn": "<dynamodb_table_arn from terraform outputs>"
57+
},
58+
"private_subnets": {
59+
"arns": [<private_subnet_ids from terraform outputs>]
60+
},
61+
"vpc": {
62+
"arn": "<vpc_arn from terraform outputs>"
63+
}
64+
}
65+
}
66+
}
67+
EOF
68+
```
69+
70+
The create network request returns a `resource_id`. For example:
71+
72+
[,yaml,lines=11]
73+
----
74+
{
75+
"operation":{
76+
"id":"cpas8k6r4up5li18auh0",
77+
"metadata":{
78+
"@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateNetworkMetadata",
79+
"network_id":"cpb338gekjj5i1cpj3t0"
80+
},
81+
"state":"STATE_IN_PROGRESS",
82+
"started_at":"2024-05-28T19:33:54.631Z",
83+
"type":"TYPE_CREATE_NETWORK",
84+
"resource_id":"cpb338gekjj5i1cpj3t0"
85+
}
86+
}
87+
----
88+
89+
== Create a cluster with the Cloud API
90+
91+
To create a cluster, issue a create cluster request using the Terraform outputs.
92+
93+
For information about the fields, see
94+
https://redpanda-api-documentation.netlify.app/#post-/v1beta2/clusters[Create Cluster].
95+
96+
```
97+
curl -X POST "https://api.redpanda.com/v1beta2/clusters" \
98+
-H "accept: application/json"\
99+
-H "content-type: application/json" \
100+
-H "authorization: Bearer $YOUR_TOKEN" \
101+
-d @- << EOF
102+
{
103+
"cloud_provider":"CLOUD_PROVIDER_AWS",
104+
"connection_type":"CONNECTION_TYPE_PRIVATE",
105+
"name":"<name of cluster>",
106+
"resource_group_id":"<resource group id of the network>",
107+
"network_id":"<resource_id of network from previous step>",
108+
"region":"<region>",
109+
"throughput_tier":"<throughput tier>",
110+
"type":"TYPE_BYOC",
111+
"zones":["<zone 1>", "<zone 2>", "<zone 3>"],
112+
"redpanda_version": "<redpanda version>",
113+
"customer_managed_resources": {
114+
"aws": {
115+
"agent_instance_profile": {
116+
"arn": "<agent_instance_profile_arn from terraform outputs>"
117+
},
118+
"connectors_node_group_instance_profile": {
119+
"arn": "<connectors_node_group_instance_profile_arn from terraform outputs>"
120+
},
121+
"redpanda_node_group_instance_profile": {
122+
"arn": "<redpanda_node_group_instance_profile_arn from terraform outputs>"
123+
},
124+
"utility_node_group_instance_profile": {
125+
"arn": "<utility_node_group_instance_profile_arn from terraform outputs>"
126+
},
127+
"connectors_security_group": {
128+
"arn": "<connectors_security_group_arn from terraform outputs>"
129+
},
130+
"node_security_group": {
131+
"arn": "<node_security_group_arn from terraform outputs>"
132+
},
133+
"utility_security_group": {
134+
"arn": "<utility_security_group_arn from terraform outputs>"
135+
},
136+
"redpanda_agent_security_group": {
137+
"arn": "<redpanda_agent_security_group_arn from terraform outputs>"
138+
},
139+
"redpanda_node_group_security_group": {
140+
"arn": "<redpanda_node_group_security_group_arn from terraform outputs>"
141+
},
142+
"cluster_security_group": {
143+
"arn": "<cluster_security_group_arn from terraform outputs>"
144+
},
145+
"k8s_cluster_role": {
146+
"arn": "<k8s_cluster_role_arn from terraform outputs>"
147+
},
148+
"cloud_storage_bucket": {
149+
"arn": "<cloud_storage_bucket_arn from terraform outputs>"
150+
},
151+
"permissions_boundary_policy": {
152+
"arn": "<permissions_boundary_policy_arn from terraform outputs>"
153+
}
154+
}
155+
},
156+
# <The following aws_private_link section is optional. For more information, see https://docs.redpanda.com/current/deploy/deployment-option/cloud/aws-privatelink/.>
157+
"aws_private_link": {
158+
"enabled": true,
159+
"allowed_principals": [<allowed principals>],
160+
"connect_console": <true|false>
161+
}
162+
}
163+
EOF
164+
```
165+
166+
The create cluster request returns a `resource_id`, which is required in the next step. For example:
167+
168+
```bash
169+
{
170+
"operation":{
171+
"id":"cpas8k6r4up5li18auhg",
172+
"metadata":{
173+
"@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateClusterMetadata",
174+
"cluster_id":"cpb33c8ekjj5i1cpj3v0"
175+
},
176+
"state":"STATE_IN_PROGRESS",
177+
"started_at":"2024-05-28T19:34:09.501Z",
178+
"type":"TYPE_CREATE_CLUSTER",
179+
"resource_id":"cpb33c8ekjj5i1cpj3v0"
180+
}
181+
}
182+
```
183+
184+
== Create cluster resources
185+
186+
To create the initial cluster resources, run `rpk cloud byoc aws apply`. This creates an autoscaling group, an agent VM, and the following resources:
187+
188+
* S3 objects
189+
* Launch template
190+
* Autoscaling group
191+
192+
NOTE: You must have the `iam_rpk_user.tf` permissions described in the prerequisites.
193+
194+
```bash
195+
rpk cloud login \
196+
--save \
197+
--client-id='<client-id>’ \
198+
--client-secret='<client-secret>' \
199+
--no-profile
200+
201+
rpk cloud byoc aws apply \
202+
--redpanda-id='<resource_id of cluster from previous step>'
203+
```
204+
205+
Output:
206+
207+
```bash
208+
Checking RPK User... PASSED
209+
Checking IAM Instance Profiles... PASSED
210+
Checking Storage... PASSED
211+
Checking Network... PASSED
212+
Reconciling agent infrastructure...
213+
Running apply {"provisioner": "redpanda-bootstrap"}
214+
Finished apply {"provisioner": "redpanda-bootstrap"}
215+
Running apply {"provisioner": "redpanda-network"}
216+
Finished apply {"provisioner": "redpanda-network"}
217+
Running apply {"provisioner": "redpanda-agent"}
218+
Finished apply {"provisioner": "redpanda-agent"}
219+
The Redpanda cluster is deploying. This can take up to 45 minutes. View status at https://cloud.redpanda.com/clusters/<resource_id of cluster from previous step>/overview.
220+
```
221+
222+
The agent VM now is running and handles the remaining provisioning steps. This can take up to 45 minutes. When provisioning completes, the cluster status updates to `Running`. If the cluster remains in `Creating` status after 45 minutes, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^].
223+
224+
=== Validatation checks
225+
226+
The `rpk cloud byoc aws apply` command performs validation checks before proceeding with provisioning:
227+
228+
* RPK user: Checks if the user running the command has sufficient privileges to provision the agent. Any
229+
missing permissions are displayed in the output.
230+
231+
* IAM instance profile: Checks that `connectors_node_group_instance_profile`, `redpanda_node_group_instance_profile`,
232+
`utility_node_group_instance_profile`, and `k8s_cluster_role` have the minimum required permissions. Any missing permissions are displayed in the output.
233+
234+
* Storage: Checks that the `management_bucket` exists and is versioned, checks that the `cloud_storage_bucket` exists and is not versioned, and checks that the `dynamodb_table` exists.
235+
236+
* Network: Checks that the VPC exists, checks that the subnets exist and have the expected tags, and checks that the security groups exist and have the desired ingress and egress rules.
237+
238+
If you think validation errors are erroneous, you can rerun the command with the `--no-validate` tag
239+
to skip validation.
240+
241+
== Check cluster status
242+
243+
You can check the cluster status with the Cloud API or the Redpanda Cloud UI.
244+
245+
Example using the returned `operation_id`:
246+
247+
```bash
248+
curl -X GET "https://api.redpanda.com/v1beta2/operations/<operation_id of operation from previous step>" \
249+
-H "accept: application/json"\
250+
-H "content-type: application/json" \
251+
-H "authorization: Bearer $YOUR_TOKEN"
252+
```
253+
254+
Example retrieving cluster:
255+
256+
```bash
257+
curl -X GET "https://api.redpanda.com/v1beta2/clusters/<resource_id of cluster from previous step>" \
258+
-H "accept: application/json"\
259+
-H "content-type: application/json" \
260+
-H "authorization: Bearer $YOUR_TOKEN"
261+
```

‎modules/get-started/pages/cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`.
66

7-
NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the most security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[customer-managed VPC on GCP].
7+
NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[customer-managed VPC on GCP].
88

99
See also: xref:get-started:cloud-overview.adoc#redpanda-cloud-architecture[Redpanda Cloud architecture].
1010

‎modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
include::shared:partial$feature-flag.adoc[]
66

7-
With a standard BYOC cluster, Redpanda manages the VPC lifecycle. For additional security, you can deploy the Redpanda data plane into your existing shared VPC and manage the VPC lifecycle yourself. When you create a BYOC cluster, you specify your VPC and service account. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With a *customer-managed* VPC:
7+
With a standard BYOC cluster, Redpanda manages the VPC lifecycle. For additional security, you can deploy the Redpanda data plane into your existing shared VPC and manage the VPC lifecycle yourself. When you create a BYOC cluster, you specify your VPC and service account. The Redpanda Cloud agent does not create a VPC or network resources. With a *customer-managed* VPC:
88

99
* You provide your own VPC in your Google Cloud account.
1010
* You maintain more control of your Google Cloud account, because Redpanda requires fewer permissions than standard BYOC clusters.

‎modules/get-started/pages/whats-new-cloud.adoc‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,13 @@ This page lists new features added in Redpanda Cloud.
99

1010
== November 2024
1111

12-
=== Customer-managed VNet on Azure
12+
=== Customer-managed VPC on AWS: beta
1313

14-
With standard BYOC clusters, Redpanda manages security policies and resources for your virtual network (VNet), including subnetworks, managed identities, IAM roles, security groups, and storage accounts. For the highest level of security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/azure/vnet-azure.adoc[customer-managed VNet on Azure].
14+
With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc[customer-managed VPC on AWS].
15+
16+
=== Customer-managed VNet on Azure: LA
17+
18+
With standard BYOC clusters, Redpanda manages security policies and resources for your virtual network (VNet), including subnetworks, managed identities, IAM roles, security groups, and storage accounts. For the highest level of security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/azure/vnet-azure.adoc[customer-managed VNet on Azure]. Because Azure functionality is provided in limited availability, to unlock this feature, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^].
1519

1620
== October 2024
1721

0 commit comments

Comments
 (0)