How to pass in secrets to cluster config (#283)

* Add secrets management content

* Revert to original include directive

* Add manage secrets to What's new

* Cross reference doc for setting config property

* Apply suggestions from automated review

* Correct notation for referencing secret in API

* Explicitly mention that Control Plane API is first called to retrieve URL

* Update secret might take a while to process

* Apply suggestions

* Apply suggestions from code review

Co-authored-by: Michele Cyran <[email protected]>

* Update local-antora-playbook.yml

* Update local-antora-playbook.yml

---------

Co-authored-by: Michele Cyran <[email protected]>

Author: kbatuigas

modules/get-started/pages/whats-new-cloud.adoc

@@ -29,7 +29,18 @@ Iceberg topics are supported for BYOC clusters in AWS and GCP.
 
 You can now xref:manage:cluster-maintenance/config-cluster.adoc[configure certain cluster properties] with `rpk cluster config` or with the Cloud API. For example, you can enable and manage xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg topics], xref:develop:data-transforms/index.adoc[data transforms], and xref:manage:audit-logging.adoc[audit logging]. Available properties are listed in xref:reference:properties/cluster-properties.adoc[Cluster Configuration Properties].
 
-Iceberg topics properties are available for clusters running Redpanda version 25.1 or later. 
+Iceberg topics properties are available for clusters running Redpanda version 25.1 or later.
+
+=== Manage secrets for cluster configuration
+
+Redpanda Cloud now supports managing secrets that you can reference in cluster properties, for example, to configure Iceberg topics. You can create, update, and delete secrets and reference a secret in cluster properties using `rpk` or the Cloud API.
+
+See also:
+
+* Manage secrets using xref:reference:rpk/rpk-security/rpk-security-secret.adoc[`rpk security secret`]
+* Manage secrets using the xref:manage:api/cloud-dataplane-api.adoc#manage-secrets[Data Plane API]
+* Reference a secret in a cluster property using xref:reference:rpk/rpk-cluster/rpk-cluster-config-set.adoc[`rpk cluster config set`]
+* Reference a secret in a cluster property using the xref:manage:cluster-maintenance/config-cluster.adoc[Control Plane API]
 
 === Data transforms: GA
 

modules/manage/pages/api/cloud-dataplane-api.adoc

@@ -15,13 +15,13 @@ The xref:manage:api/cloud-api-overview.adoc#cloud-api-architecture[data plane] c
 BYOC or Dedicated::
 +
 --
-To retrieve the Data Plane API URL of a cluster, make a request to xref:api:ROOT:cloud-controlplane-api.adoc#get-/v1/clusters/-id-[`GET /v1/clusters/\{id}`].
+To retrieve the Data Plane API URL of a cluster, make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#get-/v1/clusters/-id-[`GET /v1/clusters/\{id}`] endpoint of the Control Plane API.
 --
 
 Serverless::
 +
 --
-To retrieve the Data Plane API URL of a cluster, make a request to xref:api:ROOT:cloud-controlplane-api.adoc#get-/v1/serverless/clusters/-id-[`GET /v1/serverless/clusters/\{id}`].
+To retrieve the Data Plane API URL of a cluster, make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#get-/v1/serverless/clusters/-id-[`GET /v1/serverless/clusters/\{id}`] endpoint of the Control Plane API.
 --
 ======
 
@@ -100,6 +100,83 @@ curl -X POST "<dataplane-api-url>/v1/topics" \
  -d '{"name":"<topic-name>"}'
 ----
 
+=== Manage secrets
+
+Secrets are stored externally in your cloud provider’s secret management service. Redpanda fetches the secrets when you reference them in cluster properties.
+
+==== Create a secret
+
+Make a request to xref:api:ROOT:cloud-dataplane-api.adoc#post-/v1/secrets[`POST /v1/secrets`]. You must use a Base64-encoded secret.
+
+[,bash]
+----
+curl -X POST "https://<dataplane-api-url>/v1/secrets" \
+ -H "accept: application/json" \
+ -H "authorization: Bearer <token>" \
+ -H "content-type: application/json" \
+ -d '{"id":"<secret-name>","scopes":["SCOPE_REDPANDA_CLUSTER"],"secret_data":"<secret-value>"}' 
+----
+
+You must include the following values:
+
+- `<dataplane-api-url>`: The base URL for the Data Plane API.
+- `<token>`: The API key you generated during authentication.
+- `<secret-name>`: The name of the secret you want to add. Use only the following characters: `^[A-Z][A-Z0-9_]*$`.
+- `<secret-value>`: The Base64-encoded secret.
+- This scope: `"SCOPE_REDPANDA_CLUSTER"`.
+
+The response returns the name and scope of the secret.
+
+You can then use the Control Plane API or `rpk` to xref:manage:cluster-maintenance/config-cluster.adoc[set a cluster property value] to reference a secret, using the secret name.
+
+For the Control Plane API, you must use the following notation with the secret name in the request body to correctly reference the secret:
+
+```bash
+"iceberg_rest_catalog_client_secret": "${secrets.<secret-name>}"
+```
+
+==== Update a secret
+
+Make a request to xref:api:ROOT:cloud-dataplane-api.adoc#put-/v1/secrets/-id-[`PUT /v1/secrets/\{id}`]. You can only update the secret value, not its name. You must use a Base64-encoded secret.
+
+[,bash]
+----
+curl -X PUT "https://<dataplane-api-url>/v1/secrets/<secret-name>" \
+ -H "accept: application/json" \
+ -H "authorization: Bearer <token>" \
+ -H "content-type: application/json" \
+ -d '{"scopes":["SCOPE_REDPANDA_CLUSTER"],"secret_data":"<new-secret-value>"}'
+----
+
+You must include the following values:
+
+- `<dataplane-api-url>`: The base URL for the Data Plane API.
+- `<secret-name>`: The name of the secret you want to update. The secret's name is also its ID.
+- `<token>`: The API key you generated during authentication.
+- This scope: `"SCOPE_REDPANDA_CLUSTER"`.
+- `<new-secret-value>`: Your new Base64-encoded secret.
+
+The response returns the name and scope of the secret. It might take several minutes for the new secret value to propagate to any cluster properties that reference it.
+
+==== Delete a secret
+
+Before you delete a secret, make sure that you remove references to it from your cluster configuration. 
+
+Make a request to xref:api:ROOT:cloud-dataplane-api.adoc#delete-/v1/secrets/-id-[`DELETE /v1/secrets/\{id}`].
+
+[,bash]
+----
+curl -X DELETE "https://<dataplane-api-url>/v1/secrets/<secret-name>" \
+ -H "accept: application/json" \
+ -H "authorization: Bearer <token>" \
+----
+
+You must include the following values:
+
+- `<dataplane-api-url>`: The base URL for the Data Plane API.
+- `<secret-name>`: The name of the secret you want to delete.
+- `<token>`: The API key you generated during authentication.
+
 === Use Redpanda Connect
 
 Use the API to manage xref:develop:connect/about.adoc[Redpanda Connect pipelines] in Redpanda Cloud.