[25.3] Import mode for Schema Registry (#1391)

Author: kbatuigas

modules/get-started/pages/release-notes/redpanda.adoc

@@ -7,68 +7,9 @@ This topic includes new content added in version {page-component-version}. For a
 * xref:redpanda-cloud:get-started:whats-new-cloud.adoc[]
 * xref:redpanda-cloud:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility]
 
-== Iceberg topics with AWS Glue
+== Schema Registry import mode
 
-A new xref:manage:iceberg/iceberg-topics-aws-glue.adoc[integration with AWS Glue Data Catalog] allows you to add Redpanda topics as Iceberg tables in your data lakehouse. The AWS Glue catalog integration is available in Redpanda version 25.1.7 and later.
+Redpanda Schema Registry now supports an import mode that allows you to import existing schemas and retain their current IDs and version numbers. Import mode is useful when migrating from another schema registry. 
 
-See xref:manage:iceberg/rest-catalog/index.adoc[] for supported Iceberg REST catalog integrations.
-
-== JSON Schema support for Iceberg topics
-
-Redpanda now supports JSON Schema for Iceberg topics. This allows you to use all supported schema types (Protobuf, Avro, and JSON Schema) for Iceberg topics. For more information, see xref:manage:iceberg/specify-iceberg-schema.adoc[].
-
-== Manage SASL users with Kafka APIs
-
-Redpanda now supports the following Kafka APIs for managing SASL user credentials as described in https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API[KIP-554^]:
-
-- DescribeUserSCRAMCredentialsRequest (requires Cluster Describe ACLs)
-- AlterUserSCRAMCredentialsRequest (requires Cluster Modify ACLs)
-
-See also: xref:manage:security/authentication.adoc#sasl[Configure Authentication] and xref:manage:security/authorization/acl.adoc[]
-
-== Schema Registry Authorization
-
-You can now use the Enterprise-licensed feature xref:manage:schema-reg-authorization.adoc[Schema Registry Authorization] to control access to Schema Registry subjects and operations using either `rpk` or the link:/api/doc/schema-registry/operation/operation-get_security_acls[Redpanda Schema Registry API] endpoints. Schema Registry Authorization offers more granular control over who can do what with your Redpanda Schema Registry resources. ACLs used for Schema Registry access also support RBAC roles.
-
-== Retrieve serialized Protobuf schemas with Schema Registry API
-
-Starting in version 25.2, the Schema Registry API supports retrieving serialized schemas (Protobuf only) using the `format=serialized` query parameter for the following endpoints:
-
-- `GET /schemas/ids/\{id}`
-- `POST /subjects/\{subject}`
-- `GET subjects/\{subject}/versions/\{version}`
-- `GET subjects/\{subject}/versions/\{version}/schema`
-
-This helps facilitate migration of Protobuf clients to Redpanda. See the link:/api/doc/schema-registry[Schema Registry API reference] for details.
-
-== Support for confluent-kafka-javascript client
-
-The `confluent-kafka-javascript` client is now validated with Redpanda. For a list of validated clients, see xref:develop:kafka-clients.adoc[].
-
-== HTTP Proxy authentication changes
-
-include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[tags=!*;include-config-link]
-
-If you need to maintain the current HTTP Proxy functionality while transitioning to authenticated clients, configure the following HTTP Proxy client properties in your `redpanda.yaml` configuration:
-
-- xref:reference:properties/broker-properties.adoc#scram_username[`scram_username`]: Username for SASL/SCRAM authentication
-- xref:reference:properties/broker-properties.adoc#scram_password[`scram_password`]: Password for SASL/SCRAM authentication
-- xref:reference:properties/broker-properties.adoc#sasl_mechanism[`sasl_mechanism`]: SASL mechanism (typically `SCRAM-SHA-256` or `SCRAM-SHA-512`)
-
-== New topic properties
-
-- xref:reference:properties/topic-properties.adoc#max.compaction.lag.ms[`max.compaction.lag.ms`]
-- xref:reference:properties/topic-properties.adoc#min.compaction.lag.ms[`min.compaction.lag.ms`]
-
-== New cluster properties
-
-- xref:reference:properties/cluster-properties.adoc#audit_failure_policy[`audit_failure_policy`]
-- xref:reference:properties/cluster-properties.adoc#iceberg_rest_catalog_base_location[`iceberg_rest_catalog_base_location`]
-- xref:reference:properties/cluster-properties.adoc#kafka_topics_max[`kafka_topics_max`]
-- xref:reference:properties/cluster-properties.adoc#log_compaction_merge_max_ranges[`log_compaction_merge_max_ranges`]
-- xref:reference:properties/cluster-properties.adoc#log_compaction_merge_max_segments_per_range[`log_compaction_merge_max_segments_per_range`]
-- xref:reference:properties/cluster-properties.adoc#log_compaction_pause_use_sliding_window[`log_compaction_pause_use_sliding_window`]
-- xref:reference:properties/cluster-properties.adoc#max_compaction_lag_ms[`max_compaction_lag_ms`]
-- xref:reference:properties/cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`]
-- xref:reference:properties/cluster-properties.adoc#schema_registry_enable_authorization[`schema_registry_enable_authorization`]
-- xref:reference:properties/cluster-properties.adoc#topic_label_aggregation_limit[`topic_label_aggregation_limit`]
+Starting with this release, import mode must be used when importing schemas. Read-write mode no longer allows specifying a schema ID and version when registering a schema.
+See xref:manage:schema-reg/schema-reg-api.adoc#set-schema-registry-mode[Use the Schema Registry API] for more information.

modules/manage/pages/schema-reg/schema-reg-api.adoc

@@ -976,52 +976,177 @@ Curl::
 --
 ====
 
-== Use READONLY mode for disaster recovery
+== Set Schema Registry mode
 
-The `/mode` endpoint allows you to put Schema Registry in read-only or read-write mode. A read-only Schema Registry does not accept direct writes. An active production cluster can replicate schemas to a read-only Schema Registry to keep it in sync, for example using Redpanda's https://github.com/redpanda-data/schema-migration/[Schema Migration tool^]. Users in the disaster recovery (DR) site cannot update schemas directly, so the DR cluster has an exact replica of the schemas in production. In a failover due to a disaster or outage, you can set Schema Registry to read-write mode, taking over for the failed cluster and ensuring availability.
+The `/mode` endpoint allows you to put Schema Registry in read-only, read-write, or import mode.
+
+* In read-write mode (the default), you can both register and look up schemas.
+* In <<use-readonly-mode-for-disaster-recovery,read-only mode>>, you can only look up schemas. This mode is most useful for standby clusters in a disaster recovery setup.
+* In <<use-import-mode-for-migration,import mode>>, you can only register schemas. This mode is most useful for target clusters in a migration setup.
 
 If authentication is enabled on Schema Registry, only superusers can change global and subject-level modes.
 
+[CAUTION]
+====
+*Breaking change in Redpanda 25.3:* In Redpanda versions before 25.3, you could specify a schema ID or version when registering a schema in read-write mode.
+
+Starting with 25.3, read-write mode returns an error when you try to register a schema with a specific ID or version. If you have custom scripts that rely on the ability to specify an ID or version with Redpanda 25.2 and earlier, you must do either of the following:
+
+* Omit the ID and version fields when registering a schema. The schema will be registered under a new ID and version.
+* Change the Schema Registry or the subject to import mode.
+====
+
 === Get global mode
 
 To link:/api/doc/schema-registry/operation/operation-get_mode[query the global mode] for Schema Registry:
 
+[tabs]
+====
+rpk::
++
+--
+```bash
+rpk registry mode get --global
+```
+--
+
+Curl::
++
 ```bash
 curl http://localhost:8081/mode
 ```
+====
 
 === Set global mode
 
+Set the mode for Schema Registry at a global level. This mode applies to all subjects that do not have a specific mode set.
+
+[tabs]
+====
+rpk::
++
+--
+```bash
+rpk registry mode set --mode <mode> --global
+```
+--
+
+Curl::
++
 ```bash
 curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": <mode>}' http://localhost:8081/mode
 ```
+====
 
 Replace the `<mode>` placeholder with the desired mode:
 
 - `READONLY`
 - `READWRITE`
+- `IMPORT`
 
 === Get mode for a subject
 
-This request returns an error if there is no specific mode set for the subject:
+To look up the mode for a specific subject:
 
+[tabs]
+====
+rpk::
++
+--
 ```bash
-curl http://localhost:8081/mode/<subject>
+rpk registry mode get <subject-name>
 ```
+--
 
-To retrieve the mode regardless of whether or not the subject has a specific mode, use the `defaultToGlobal` parameter:
-
+Curl::
++
 ```bash
 curl http://localhost:8081/mode/<subject>?defaultToGlobal=true
 ```
+====
+
+This request returns the mode that is enforced. If the subject is set to a specific mode (to override the global mode), it returns the override mode. Otherwise, it returns the global mode.
+
+To retrieve the subject-level override if it exists, use:
+
+```bash
+curl http://localhost:8081/mode/<subject>
+```
 
-This request returns the mode that is enforced. If the subject is set to a specific mode (to override the global mode), it will return the override mode, otherwise it returns the global mode.
+This request returns an error if there is no specific mode set for the subject.
 
 === Set mode for a subject
 
+[tabs]
+====
+rpk::
++
+--
+```bash
+rpk registry mode set <subject-name> --mode READONLY
+```
+--
+
+Curl::
++
 ```bash
 curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "READONLY"}' http://localhost:8081/mode/<subject>
 ```
+====
+
+=== Use READONLY mode for disaster recovery
+
+A read-only Schema Registry does not accept direct writes. An active production cluster can replicate schemas to a read-only Schema Registry to keep it in sync, for example using Redpanda's https://github.com/redpanda-data/schema-migration/[Schema Migration tool^]. Users in the disaster recovery (DR) site cannot update schemas directly, so the DR cluster has an exact replica of the schemas in production. In a failover due to a disaster or outage, you can set Schema Registry to read-write mode, taking over for the failed cluster and ensuring availability.
+
+=== Use IMPORT mode for migration
+
+Set the target Schema Registry to import mode to:
+
+- Bypass compatibility checks when registering schemas.
+- Specify a specific schema ID and version for the registered schema, so you can retain the same IDs and version from the original Schema Registry and keep topic data associated with the correct schema.
+
+To enable import mode, you must have:
+
+* Either superuser access, or a Schema Registry ACL with the `alter_configs` operation on the `registry` resource. See xref:manage:schema-reg/schema-reg-authorization.adoc#enable-schema-registry-authorization[Enable Schema Registry Authorization] to learn how to enable schema registry authorization for your cluster.
+* An empty registry or subject. That is, either no schemas have ever been registered, or you must <<hard-delete-a-schema,hard-delete>> all schemas that were registered.
++
+To bypass the check for an empty registry when setting the global mode to import:
++
+[tabs]
+====
+rpk::
++
+--
+```bash
+rpk registry mode set --mode IMPORT --global --force
+```
+--
+
+Curl::
++
+```bash
+curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "IMPORT"}' http://localhost:8081/mode?force=true
+```
+====
+
+Use import mode to register a schema with a specific ID and version:
+
+[tabs]
+====
+rpk::
++
+--
+```bash
+rpk registry schema create <subject-name> --schema order.proto --id 1 --schema-version 4
+```
+--
+
+Curl::
++
+```bash
+curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "syntax = \"proto3\";\nmessage Order {\n  string id = 1;\n}", "schemaType": "PROTOBUF", "id": 1, "version": 4}' http://localhost:8081/subjects/<subject-name>/versions
+```
+====
 
 == Retrieve serialized schemas
 

modules/upgrade/partials/incompat-changes.adoc

@@ -1,5 +1,7 @@
 === Review incompatible changes
 
+* *Breaking change in Redpanda 25.3*: Schema Registry no longer allows specifying a schema ID and version when registering a schema in read-write mode. You must use import mode to register a schema with a specific ID and version. See xref:manage:schema-reg/schema-reg-api.adoc#set-schema-registry-mode[Use the Schema Registry API] for more information.
+
 * {empty}
 +
 --