DOC-1464 Document feature Disable Kafka Connect by default (#374)

micheleRP · web-flow · commit ae820b5764fd · 2025-08-05T10:05:24.000-06:00
* DOC-1464 Document feature Disable Kafka Connect by default

* coderabbit suggestion

* edit admonition and remove unnecessary files

* remove manual steps for Azure, add to What's New, overview, Cloud API
diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc
@@ -366,6 +366,16 @@
 ** xref:develop:consume-data/index.adoc[Consume Data]
 *** xref:develop:consume-data/consumer-offsets.adoc[Consumer Offsets]
 *** xref:develop:consume-data/follower-fetching.adoc[Follower Fetching]
+** xref:develop:http-proxy.adoc[]
+** xref:develop:data-transforms/index.adoc[]
+*** xref:develop:data-transforms/how-transforms-work.adoc[Overview]
+*** xref:develop:data-transforms/build.adoc[Build]
+*** xref:develop:data-transforms/configure.adoc[Configure]
+*** xref:develop:data-transforms/deploy.adoc[Deploy]
+*** xref:develop:data-transforms/test.adoc[Test]
+*** xref:develop:data-transforms/monitor.adoc[Monitor]
+*** xref:develop:data-transforms/data-transforms.adoc[Manage]
+** xref:develop:transactions.adoc[]
 ** xref:develop:managed-connectors/index.adoc[Kafka Connect]
 *** xref:develop:managed-connectors/converters-and-serialization.adoc[Converters and serialization]
 *** xref:develop:managed-connectors/monitor-connectors.adoc[Monitor Connectors]
@@ -386,16 +396,6 @@
 *** xref:develop:managed-connectors/create-postgresql-connector.adoc[PostgreSQL (Debezium) Source Connector]
 *** xref:develop:managed-connectors/create-sqlserver-connector.adoc[SQL Server (Debezium) Source Connector]
 *** xref:develop:managed-connectors/create-snowflake-connector.adoc[Snowflake Sink Connector]
-** xref:develop:http-proxy.adoc[]
-** xref:develop:data-transforms/index.adoc[]
-*** xref:develop:data-transforms/how-transforms-work.adoc[Overview]
-*** xref:develop:data-transforms/build.adoc[Build]
-*** xref:develop:data-transforms/configure.adoc[Configure]
-*** xref:develop:data-transforms/deploy.adoc[Deploy]
-*** xref:develop:data-transforms/test.adoc[Test]
-*** xref:develop:data-transforms/monitor.adoc[Monitor]
-*** xref:develop:data-transforms/data-transforms.adoc[Manage]
-** xref:develop:transactions.adoc[]
 
 * xref:manage:index.adoc[Manage]
 ** xref:manage:cluster-maintenance/index.adoc[Cluster Maintenance]
diff --git a/modules/develop/pages/managed-connectors/converters-and-serialization.adoc b/modules/develop/pages/managed-connectors/converters-and-serialization.adoc
@@ -2,6 +2,8 @@
 :description: Use converters to handle the serialization and deserialization of data between a Redpanda topic and a managed connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/converters-and-serialization.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 Connectors are a translation layer working between Redpanda and the remote system. 
 
 For *sink* connectors the translation happens in the following phases:
diff --git a/modules/develop/pages/managed-connectors/create-gcp-bigquery-connector.adoc b/modules/develop/pages/managed-connectors/create-gcp-bigquery-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a Google BigQuery Sink Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-gcp-bigquery-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 The Google BigQuery Sink connector enables you to stream any structured data from
 Redpanda to BigQuery for advanced analytics.
 
diff --git a/modules/develop/pages/managed-connectors/create-gcs-connector.adoc b/modules/develop/pages/managed-connectors/create-gcs-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a GCS Sink Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-gcs-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 The Google Cloud Storage (GCS) Sink connector stores Redpanda messages in a Google Cloud Storage bucket.
 
 == Prerequisites
diff --git a/modules/develop/pages/managed-connectors/create-iceberg-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-iceberg-sink-connector.adoc
@@ -2,6 +2,7 @@
 :description: Use the Redpanda Cloud UI to create an Iceberg Sink Connector.
 :page-aliases: cloud:managed-connectors/create-iceberg-sink-connector.adoc, deploy:deployment-option/cloud/managed-connectors/create-iceberg-sink-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
 
 You can use the Iceberg Sink connector to accomplish the following: 
 
diff --git a/modules/develop/pages/managed-connectors/create-jdbc-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-jdbc-sink-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a JDBC Sink Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-jdbc-sink-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use a JDBC Sink connector to export structured data from Redpanda to
 a relational database.
 
diff --git a/modules/develop/pages/managed-connectors/create-jdbc-source-connector.adoc b/modules/develop/pages/managed-connectors/create-jdbc-source-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a JDBC Source Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-jdbc-source-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use a JDBC Source connector to import batches of rows from MySQL,
 PostgreSQL, SQLite, and SQL Server relational databases into Redpanda topics.
 
diff --git a/modules/develop/pages/managed-connectors/create-mmaker-checkpoint-connector.adoc b/modules/develop/pages/managed-connectors/create-mmaker-checkpoint-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a MirrorMaker2 Checkpoint Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-mmaker-checkpoint-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use the MirrorMaker2 Checkpoint connector to import consumer group offsets
 from other Kafka clusters.
 
diff --git a/modules/develop/pages/managed-connectors/create-mmaker-heartbeat-connector.adoc b/modules/develop/pages/managed-connectors/create-mmaker-heartbeat-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a MirrorMaker2 Heartbeat Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-mmaker-heartbeat-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use a MirrorMaker2 Heartbeat connector to generate heartbeat messages
 to a local cluster's `heartbeat` topic.
 
diff --git a/modules/develop/pages/managed-connectors/create-mmaker-source-connector.adoc b/modules/develop/pages/managed-connectors/create-mmaker-source-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a MirrorMaker2 Source Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-mmaker-source-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use a MirrorMaker2 Source connector to import messages from another Kafka cluster.
 You can also use it to:
 
diff --git a/modules/develop/pages/managed-connectors/create-mongodb-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-mongodb-sink-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a MongoDB Sink Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-mongodb-sink-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 The MongoDB Sink managed connector exports Redpanda structured data to a MongoDB
 database.
 
diff --git a/modules/develop/pages/managed-connectors/create-mongodb-source-connector.adoc b/modules/develop/pages/managed-connectors/create-mongodb-source-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a MongoDB Source Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-mongodb-source-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 The MongoDB Source managed connector imports collections from a MongoDB database
 into Redpanda topics.
 
diff --git a/modules/develop/pages/managed-connectors/create-mysql-source-connector.adoc b/modules/develop/pages/managed-connectors/create-mysql-source-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a MySQL (Debezium) Source Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-mysql-source-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use a MySQL (Debezium) Source connector to import a stream of changes from MySQL,
 AmazonRDS, and Amazon Aurora.
 
diff --git a/modules/develop/pages/managed-connectors/create-postgresql-connector.adoc b/modules/develop/pages/managed-connectors/create-postgresql-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a PostgreSQL (Debezium) Source Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-postgresql-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use a PostgreSQL (Debezium) Source connector to import updates to Redpanda from PostgreSQL.
 
 == Prerequisites
diff --git a/modules/develop/pages/managed-connectors/create-s3-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-s3-sink-connector.adoc
@@ -2,6 +2,7 @@
 :description: Use the Redpanda Cloud UI to create an AWS S3 Sink Connector.
 :page-aliases: cloud:managed-connectors/create-s3-sink-connector.adoc, deploy:deployment-option/cloud/managed-connectors/create-s3-sink-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
 
 The Amazon S3 Sink connector exports Apache Kafka messages to files in AWS S3 buckets.
 
diff --git a/modules/develop/pages/managed-connectors/create-snowflake-connector.adoc b/modules/develop/pages/managed-connectors/create-snowflake-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a Snowflake Sink Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-snowflake-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use the Snowflake Sink connector to ingest and store Redpanda structured data into a
 Snowflake database for analytics and decision-making.
 
diff --git a/modules/develop/pages/managed-connectors/create-sqlserver-connector.adoc b/modules/develop/pages/managed-connectors/create-sqlserver-connector.adoc
@@ -2,6 +2,8 @@
 :description: Use the Redpanda Cloud UI to create a SQL Server (Debezium) Source Connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/create-sqlserver-connector.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 You can use an SQL Server (Debezium) Source connector to import updates to Redpanda from SQL Server.
 
 == Prerequisites
diff --git a/modules/develop/pages/managed-connectors/index.adoc b/modules/develop/pages/managed-connectors/index.adoc
@@ -6,9 +6,9 @@
 Use Kafka Connect to integrate your Redpanda data with different
 data systems. As managed solutions, connectors offer a simpler way to integrate
 your data than manually creating a solution with the Kafka API. You can set up
-and manage these connectors for BYOC and Dedicated clusters in the Redpanda Cloud UI. Note that you remain responsible 
+and manage these connectors for BYOC and Dedicated clusters in the Redpanda Cloud UI or Cloud API. Note that you remain responsible 
 for monitoring, alerting, and restarting Kafka Connect connectors, as these tasks 
-are not managed or monitored by Redpanda support.
+are not managed or monitored by Redpanda Support.
 
 include::shared:partial$kafka-connect.adoc[]
 
diff --git a/modules/develop/pages/managed-connectors/monitor-connectors.adoc b/modules/develop/pages/managed-connectors/monitor-connectors.adoc
@@ -2,5 +2,6 @@
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/monitor-connectors.adoc
 :description: Use metrics to monitor the health of your Redpanda managed connectors.
 
+include::shared:partial$kafka-connect.adoc[]
 
 include::ROOT:manage:partial$monitor-connectors.adoc[tag=single-source]
diff --git a/modules/develop/pages/managed-connectors/sizing-connectors.adoc b/modules/develop/pages/managed-connectors/sizing-connectors.adoc
@@ -2,6 +2,7 @@
 :description: How to choose number of tasks to set for a connector.
 :page-aliases: cloud:managed-connectors/task-count.adoc, deploy:deployment-option/cloud/managed-connectors/sizing-connectors.adoc
 
+include::shared:partial$kafka-connect.adoc[]
 
 == Connector tasks
 When you set up a connector, its main responsibility is to validate the configuration and spawn _connector tasks_, which perform the work.
diff --git a/modules/develop/pages/managed-connectors/transforms.adoc b/modules/develop/pages/managed-connectors/transforms.adoc
@@ -2,6 +2,8 @@
 :description: Single Message Transforms (SMTs) let you modify the data and its characteristics as it passes through a connector.
 :page-aliases: deploy:deployment-option/cloud/managed-connectors/transforms.adoc
 
+include::shared:partial$kafka-connect.adoc[]
+
 Single Message Transforms (SMTs) help you modify data and its characteristics as it passes through a connector, without needing additional stream processors.
 
 Prior to using an SMT with production data, test the configuration on a smaller subset of data to verify the behavior of the SMT.
diff --git a/modules/get-started/pages/cloud-overview.adoc b/modules/get-started/pages/cloud-overview.adoc
@@ -220,7 +220,7 @@ Dedicated::
 
 xref:develop:connect/about.adoc[Redpanda Connect] is integrated into Redpanda Cloud and available as a fully-managed service. Choose from a range of connectors, processors, and other components to quickly build and deploy streaming data pipelines or AI applications from the Cloud UI or using the pass:a,m[xref:{tag-pipeline-service}[Data Plane API\]]. Comprehensive metrics, monitoring, and per pipeline scaling are also available. To start using Redpanda Connect, xref:develop:connect/connect-quickstart.adoc[try this quickstart].
 
-xref:develop:managed-connectors/index.adoc[Kafka Connect] is automatically enabled on AWS and GCP clusters. With this, there is a node running for Kafka Connect even if connectors are not used. To enable Kafka Connect on Azure clusters, see xref:get-started:cluster-types/byoc/azure/create-byoc-cluster-azure.adoc#enable-kafka-connect[Enable Kafka Connect].
+xref:develop:managed-connectors/index.adoc[Kafka Connect] is disabled by default on all new clusters. To unlock this feature for your account, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. Note that when Kafka Connect is enabled, there is a node running for Kafka Connect, even if connectors are not used.
 
 == Redpanda Cloud vs Self-Managed feature compatibility
 
diff --git a/modules/get-started/pages/cluster-types/byoc/azure/create-byoc-cluster-azure.adoc b/modules/get-started/pages/cluster-types/byoc/azure/create-byoc-cluster-azure.adoc
@@ -87,7 +87,7 @@ Confirm that the Azure subscription has enough virtual CPUs (vCPUs) per instance
 
 See the https://learn.microsoft.com/en-us/azure/quotas/view-quotas[Microsoft documentation^].
 
-NOTE: If you run Redpanda Connect pipelines, or if you <<enable-kafka-connect,enable Kafka Connect>>, Redpanda also deploys an additional node pool.
+include::shared:partial$kafka-connect.adoc[]
 
 
 === Check Azure SKU restrictions
@@ -155,30 +155,6 @@ Optionally, click *Advanced settings* to specify up to five key-value custom tag
 +
 As part of agent deployment, Redpanda assigns the permissions required to run the agent. For details about these permissions, see xref:security:authorization/cloud-iam-policies-azure.adoc[Azure IAM policies].
 
-== Enable Kafka Connect
-
-To enable xref:develop:managed-connectors/index.adoc[Kafka Connect] for clusters on Azure:
-
-. Authenticate to the Redpanda Cloud API. Follow the steps in xref:manage:api/cloud-api-authentication.adoc[].
-
-. Enable Kafka Connect by making a xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters`] request:
-+
-[,bash]
-----
-curl -X PATCH "https://api.redpanda.com/v1/clusters/<cluster-id>" \
- -H 'Accept: application/json' \
- -H 'Authorization: Bearer <token>' \
- -H 'Content-Type: application/json' \
- -d '{"kafka_connect": {"enabled": true}}'
-----
-+
-Replace the following placeholders:
-+
-- `<cluster-id>`: Enter the cluster ID listed in the Redpanda Cloud UI. Go to the Cluster Overview page, and look in the Details section.
-- `<token>`: Enter the API token you received in step 1.
-
-include::get-started:partial$no-access.adoc[]
-
 == Next steps
 
 * xref:networking:azure-private-link.adoc[Configure Azure Private Link]
diff --git a/modules/get-started/pages/cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc b/modules/get-started/pages/cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc
@@ -111,7 +111,7 @@ curl -v -X PATCH \
 
 . Check Redpanda Connect is available in the Cloud UI. 
 .. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
-.. Go to the **Connect** page and you should see the **Redpanda Connect** tab alongside **Kafka Connect**. 
+.. Go to the **Connect** page and you should see Redpanda Connect. 
 
 == Next steps
 
diff --git a/modules/get-started/pages/cluster-types/create-dedicated-cloud-cluster.adoc b/modules/get-started/pages/cluster-types/create-dedicated-cloud-cluster.adoc
@@ -27,8 +27,6 @@ Private networks require either a VPC peering connection or a private connectivi
 +
 After the cluster is created, you can select the cluster on the *Clusters* page to see the overview for it.
 
-include::shared:partial$kafka-connect.adoc[]
-
 == Start streaming: example
 
 Use `rpk`, Redpanda's CLI, to build a basic streaming application that creates a topic, produces messages to it, and consumes messages from it. To learn about `rpk`, see the xref:manage:rpk/intro-to-rpk.adoc[].
diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc
@@ -15,7 +15,11 @@ xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg topics] are now generally
 
 === BYOC on Azure: GA
 
-xref:get-started:cluster-types/byoc/azure/create-byoc-cluster-azure.adoc[BYOC for Azure] is now generally available (GA). 
+xref:get-started:cluster-types/byoc/azure/create-byoc-cluster-azure.adoc[BYOC for Azure] is now generally available (GA).
+
+=== Kafka Connect disabled on new clusters
+
+xref:develop:managed-connectors/index.adoc[Kafka Connect] is now disabled by default on all new clusters. To unlock this feature for your account, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. Note that when Kafka Connect is enabled, there is a node running for Kafka Connect, even if connectors are not used.
 
 === Allowlist NAT gateway IP 
 
diff --git a/modules/manage/pages/api/cloud-dataplane-api.adoc b/modules/manage/pages/api/cloud-dataplane-api.adoc
@@ -226,6 +226,8 @@ curl -X PUT "https://api.redpanda.com/v1/redpanda-connect/pipelines/" \
 
 Use the API to configure your xref:develop:managed-connectors/index.adoc[Kafka Connect] clusters.
 
+include::shared:partial$kafka-connect.adoc[]
+
 NOTE: Kafka Connect is supported in BYOC and Dedicated clusters only.
 
 ==== Create a Kafka Connect cluster secret
diff --git a/modules/manage/partials/monitor-connectors.adoc b/modules/manage/partials/monitor-connectors.adoc
diff --git a/modules/shared/partials/community-supported-connectors.adoc b/modules/shared/partials/community-supported-connectors.adoc
diff --git a/modules/shared/partials/kafka-connect.adoc b/modules/shared/partials/kafka-connect.adoc
