BYOVPC support for secrets and Iceberg REST catalogs (#313)

kbatuigas · Feediver1 · web-flow · commit f4ea90758805 · 2025-06-05T18:28:10.000-07:00
* Add operator and modified cluster service commands for GCP BYOVPC

* Add note for AWS clusters

* Add new doc for enabling secrets management

* Using secrets in cluster properties requires specific notation

* Add new doc for enabling secrets GCP BYOVPC to nav tree

* Since secrets and Iceberg cluster configs are self serve, move beta note to About doc instead

* Use single sourced branch to preview BYOVPC updates

* Minor edits'

* Fix API xref

* Rephrase per SME feedback

* Add to What's New

* Update local-antora-playbook.yml

---------

Co-authored-by: Joyce Fee &lt;102751339+Feediver1@users.noreply.github.com&gt;
diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc
@@ -19,6 +19,7 @@
 **** xref:get-started:cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc[]
 **** xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[]
 **** xref:get-started:cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc[Enable Redpanda Connect on a BYOVPC Cluster on GCP]
+**** xref:get-started:cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc[Enable Secrets Management on a BYOVPC Cluster on GCP]
 *** xref:get-started:cluster-types/byoc/remote-read-replicas.adoc[]
 ** xref:get-started:cluster-types/create-dedicated-cloud-cluster.adoc[]
 
diff --git a/modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc b/modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc
@@ -18,6 +18,13 @@ When you create a BYOCVPC cluster, you specify your VPC and service account. The
 
 The https://github.com/redpanda-data/cloud-examples/tree/main/customer-managed/aws[Redpanda Cloud Examples repository^] contains https://developer.hashicorp.com/terraform[Terraform^] code that deploys the resources required for a BYOVPC cluster on AWS. You'll need to create these resources in advance and give them to Redpanda during cluster creation. Variables are provided in the code so you can exclude resources that already exist in your environment, such as the VPC.
 
+[NOTE]
+====
+Secrets management is enabled by default with the Terraform code in the example repository. It allows you to store and read secrets in your cluster, for example to integrate a REST catalog with Iceberg-enabled topics.
+
+For existing BYOVPC clusters, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^] to enable secrets management.
+====
+
 == Prerequisites
 
 * Access to an AWS project in which you create your cluster.
@@ -58,7 +65,7 @@ You can update the example configuration to customize your setup:
 
 - Enable PrivateLink (`"enable_private_link": true`).
 - Preserve cluster data when deleting the cluster (`"force_destroy_cloud_storage": false`).
-- Redpanda Connect is enabled by default. To disable Redpanda Connect, set `"enable_redpanda_connect": false` in the `byoc.auto.tfvars.json` file, and remove the additional security groups and node instance profiles that are required for Redpanda Connect. 
+- Redpanda Connect is enabled by default. To disable Redpanda Connect, set `"enable_redpanda_connect": false` in the `byoc.auto.tfvars.json` file, and remove the additional security groups and node instance profiles that are required for Redpanda Connect.
 - Use https://docs.aws.amazon.com/IAM/latest/UserGuide/access_iam-tags.html[condition tags^] to control resource modifications based on AWS tags. For example:
 +
 ```json
diff --git a/modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc b/modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc
@@ -0,0 +1,120 @@
+= Enable Secrets Management on an Existing BYOVPC Cluster on GCP
+:description: Store and read secrets in your existing BYOVPC cluster.
+:page-beta: true
+
+[IMPORTANT]
+====
+BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or https://www.redpanda.com/price-estimator[Redpanda Sales^].
+==== 
+
+Storing secrets in your cluster allows you to keep your cloud infrastructure secure as you integrate your data across different systems, for example, REST catalogs with your Iceberg-enabled topics. If you do not have secrets management enabled on an existing BYOVPC cluster, you can do so by following the steps on this page to update your cluster configuration. You can also create xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[a new BYOVPC cluster] with secrets management already enabled.
+
+Replace all `<placeholders>` with your own values.
+
+. Create one new service account with the necessary permissions and roles. 
++
+.Show commands
+[%collapsible]
+====
+```bash
+# Account used to check for and read secrets
+
+gcloud iam service-accounts create redpanda-operator \
+  --display-name="Redpanda Operator Service Account"
+
+cat << EOT > redpanda-operator.role
+{
+  "name": "redpanda_operator_role",
+  "title": "Redpanda Operator Role",
+  "description": "Redpanda Operator Role",
+  "includedPermissions": [
+    "resourcemanager.projects.get",
+    "secretmanager.secrets.get",
+    "secretmanager.versions.access"
+  ],
+}
+EOT
+
+gcloud iam roles create redpanda_operator_role --project=<service-project-id> --file redpanda-operator.role
+
+gcloud projects add-iam-policy-binding <service-project-id> \
+  --member="serviceAccount:redpanda-operator@<service-project-id>.iam.gserviceaccount.com" \
+  --role="projects/<service-project-id>/roles/redpanda_operator_role"
+```
+====
+
+. Update the existing Redpanda cluster service account with the necessary permissions to read secrets.
++
+.Show commands
+[%collapsible]
+====
+```bash
+cat << EOT > redpanda-cluster.role
+{
+  "name": "redpanda_cluster_role",
+  "title": "Redpanda Cluster Role",
+  "description": "Redpanda Cluster Role",
+  "includedPermissions": [
+    "resourcemanager.projects.get",
+    "secretmanager.secrets.get",
+    "secretmanager.versions.access"
+  ],
+}
+EOT
+
+gcloud iam roles create redpanda_cluster_role --project=<service-project-id> --file redpanda-cluster.role
+
+gcloud projects add-iam-policy-binding <service-project-id> \
+  --member="serviceAccount:redpanda-cluster@<service-project-id>.iam.gserviceaccount.com" \
+  --role="projects/<service-project-id>/roles/redpanda_cluster_role"
+```
+====
+
+. Bind the new service account. 
++
+The account ID of the GCP service account is used to configure service account bindings. This account ID is the local part of the email address for the GCP service account. For example, if the GCP service account is `my-gcp-sa@my-project.iam.gserviceaccount.com`, then the account ID is `my-gcp-sa`.
++
+.Show commands
+[%collapsible]
+====
+```
+gcloud iam service-accounts add-iam-policy-binding <redpanda_operator-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
+    --role roles/iam.workloadIdentityUser \
+    --member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-system/<redpanda_operator-gcp-sa-account-id>]"
+```
+====
+
+. Make a xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/\{cluster-id}`] request to update the cluster configuration.
++
+.Show request
+[%collapsible]
+====
+```bash
+export CLUSTER_PATCH_BODY=`cat << EOF
+{
+    "customer_managed_resources": {
+        "gcp": {
+            "redpanda_operator_service_account": {
+              "email": "<redpanda_operator-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com"
+            }
+        }
+    }
+}
+EOF`
+curl -v -X PATCH \
+-H "Content-Type: application/json" \
+-H "Authorization: Bearer $AUTH_TOKEN" \
+-d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/<cluster-id>
+```
+====
+
+[[check-secrets-cloud-ui]]
+. Check secrets management is available in the Cloud UI. 
+.. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
+.. Go to the **Secrets Store** page of your cluster. You should be able to create a new secret.
+
+== Next steps
+
+* xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[Reference a secret in a cluster property].
+* xref:manage:iceberg/use-iceberg-catalogs.adoc[Integrate a catalog] for querying Iceberg topics in your cluster.
+
diff --git a/modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc b/modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc
@@ -280,12 +280,61 @@ gcloud projects add-iam-policy-binding <host-project-id> \
 [%collapsible]
 ====
 ```bash
+cat << EOT > redpanda-cluster.role
+{
+  "name": "redpanda_cluster_role",
+  "title": "Redpanda Cluster Role",
+  "description": "Redpanda Cluster Role",
+  "includedPermissions": [
+    "resourcemanager.projects.get",
+    "secretmanager.secrets.get",
+    "secretmanager.versions.access"
+  ],
+}
+EOT
+
 gcloud iam service-accounts create redpanda-cluster \
   --display-name="Redpanda Cluster Service Account"
 
 gcloud storage buckets add-iam-policy-binding gs://<tiered-storage-bucket-name> \
   --member="serviceAccount:redpanda-cluster@<service-project-id>.iam.gserviceaccount.com" \
   --role="roles/storage.objectAdmin"
+
+gcloud iam roles create redpanda_cluster_role --project=<service-project-id> --file redpanda-cluster.role
+
+gcloud projects add-iam-policy-binding <service-project-id> \
+  --member="serviceAccount:redpanda-cluster@<service-project-id>.iam.gserviceaccount.com" \
+  --role="projects/<service-project-id>/roles/redpanda_cluster_role"
+```
+====
+
+* Redpanda operator service account
++
+.Show commands
+[%collapsible]
+====
+```bash
+gcloud iam service-accounts create redpanda-operator \
+  --display-name="Redpanda Operator Service Account"
+
+cat << EOT > redpanda-operator.role
+{
+  "name": "redpanda_operator_role",
+  "title": "Redpanda Operator Role",
+  "description": "Redpanda Operator Role",
+  "includedPermissions": [
+    "resourcemanager.projects.get",
+    "secretmanager.secrets.get",
+    "secretmanager.versions.access"
+  ],
+}
+EOT
+
+gcloud iam roles create redpanda_operator_role --project=<service-project-id> --file redpanda-operator.role
+
+gcloud projects add-iam-policy-binding <service-project-id> \
+  --member="serviceAccount:redpanda-operator@<service-project-id>.iam.gserviceaccount.com" \
+  --role="projects/<service-project-id>/roles/redpanda_operator_role"
 ```
 ====
 
@@ -536,6 +585,18 @@ gcloud iam service-accounts add-iam-policy-binding <redpanda-cluster-gcp-sa-acco
 ```
 ====
 
+* Redpanda operator service account
++
+.Show command
+[%collapsible]
+====
+```
+gcloud iam service-accounts add-iam-policy-binding <redpanda_operator-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
+    --role roles/iam.workloadIdentityUser \
+    --member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-system/<redpanda_operator-gcp-sa-account-id>]"
+```
+====
+
 * Redpanda Console service account
 +
 .Show command
diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc
@@ -15,6 +15,12 @@ Serverless clusters now support the following new xref:reference:tiers/serverles
 
 == May 2025
 
+=== Secrets management for BYOVPC clusters on AWS and GCP
+
+You can now create new BYOVPC clusters with secrets management enabled by default on xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc[AWS] and xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[GCP].
+
+You can also enable secrets management for existing BYOVPC clusters on AWS and GCP. For GCP, see xref:get-started:cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc[Enable Secrets Management for BYOVPC Clusters on GCP]. For AWS, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^].
+
 === Serverless Standard: deprecated
 
 Serverless Standard is deprecated. All existing clusters will be migrated to the new xref:get-started:cluster-types/serverless.adoc[Serverless] platform (with higher usage limits, 99.9% SLA, and additional regions) on August 31, 2025.
diff --git a/modules/manage/pages/cluster-maintenance/config-cluster.adoc b/modules/manage/pages/cluster-maintenance/config-cluster.adoc
@@ -35,6 +35,13 @@ For example, to enable data transforms, set xref:reference:properties/cluster-pr
 rpk cluster config set data_transforms_enabled true
 ----
 
+To set a cluster property with a secret, you must use the following notation:
+
+[source,bash]
+----
+rpk cluster config set iceberg_rest_catalog_client_secret ${secrets.<secret-name>}
+----
+
 NOTE: Some properties require a rolling restart, and it can take several minutes for the update to complete. The `rpk cluster config set` command returns the operation ID.  
 
 
@@ -72,6 +79,22 @@ curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \
 
 The xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /clusters/{cluster.id}`] request returns the ID of a long-running operation. You can check the status of the operation by polling the xref:api:ROOT:cloud-controlplane-api.adoc#get-/v1/operations/-id-[`GET /operations/\{id}`] endpoint.
 
+To set a cluster property with a secret, you must use the following notation with the secret name:
+
+[source,bash]
+----
+curl -H "Authorization: Bearer <token>" -X PATCH \
+"https://api.cloud.redpanda.com/v1/clusters/<cluster-id>" \
+  -H 'accept: application/json'\
+  -H 'content-type: application/json' \
+  -d '{"cluster_configuration": {
+        "custom_properties": {
+            "iceberg_rest_catalog_client_secret": "${secrets.<secret-name>}"
+            }
+        }
+    }'
+----
+
 NOTE: Some properties require a rolling restart for the update to take effect. This triggers a xref:manage:api/cloud-byoc-controlplane-api.adoc#lro[long-running operation] that can take several minutes to complete.
 
 --
diff --git a/modules/manage/pages/iceberg/about-iceberg-topics.adoc b/modules/manage/pages/iceberg/about-iceberg-topics.adoc
@@ -2,4 +2,6 @@
 :description: Learn how Redpanda can integrate topics with Apache Iceberg.
 :page-beta: true
 
+NOTE: The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments.
+
 include::ROOT:manage:partial$iceberg/about-iceberg-topics.adoc[]
diff --git a/modules/manage/partials/controlplane-api.adoc b/modules/manage/partials/controlplane-api.adoc
@@ -307,7 +307,7 @@ endif::[]
 
 == Update cluster configuration
 
-To update your cluster configuration properties, make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-id-[`PATCH /v1/clusters/\{id}`] endpoint, passing the cluster ID as a parameter. Include the properties to update in the request body.
+To update your cluster configuration properties, make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/\{id}`] endpoint, passing the cluster ID as a parameter. Include the properties to update in the request body.
 
 ifdef::env-byoc[]
 [,bash]
