[25.3] Import mode for Schema Registry #1391
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -976,52 +976,177 @@ Curl:: | |
| -- | ||
| ==== | ||
|
|
||
| == Set Schema Registry mode | ||
|
|
||
| The `/mode` endpoint allows you to put Schema Registry in read-only, read-write, or import mode. | ||
|
|
||
| * In read-write mode (the default), you can both register and look up schemas. | ||
| * In <<use-readonly-mode-for-disaster-recovery,read-only mode>>, you can only look up schemas. This mode is most useful for standby clusters in a disaster recovery setup. | ||
| * In <<use-import-mode-for-migration,import mode>>, you can only register schemas. This mode is most useful for target clusters in a migration setup. | ||
|
|
||
| If authentication is enabled on Schema Registry, only superusers can change global and subject-level modes. | ||
|
|
||
| [CAUTION] | ||
| ==== | ||
| *Breaking change in Redpanda 25.3:* In Redpanda versions before 25.3, you could specify a schema ID or version when registering a schema in read-write mode. | ||
|
|
||
| Starting with 25.3, read-write mode returns an error when you try to register a schema with a specific ID or version. If you have custom scripts that rely on the ability to specify an ID or version with Redpanda 25.2 and earlier, you must do either of the following: | ||
|
|
||
| * Omit the ID and version fields when registering a schema. The schema will be registered under a new ID and version. | ||
| * Change the Schema Registry or the subject to import mode. | ||
| ==== | ||
|
|
||
| === Get global mode | ||
|
|
||
| To link:/api/doc/schema-registry/operation/operation-get_mode[query the global mode] for Schema Registry: | ||
|
|
||
| [tabs] | ||
| ==== | ||
| rpk:: | ||
| + | ||
| -- | ||
| ```bash | ||
| rpk registry mode get --global | ||
| ``` | ||
| -- | ||
|
|
||
| Curl:: | ||
| + | ||
| ```bash | ||
| curl http://localhost:8081/mode | ||
| ``` | ||
| ==== | ||
|
|
||
| === Set global mode | ||
|
|
||
| Set the mode for Schema Registry at a global level. This mode applies to all subjects that do not have a specific mode set. | ||
|
|
||
| [tabs] | ||
| ==== | ||
| rpk:: | ||
| + | ||
| -- | ||
| ```bash | ||
| rpk registry mode set --mode <mode> --global | ||
| ``` | ||
| -- | ||
|
|
||
| Curl:: | ||
| + | ||
| ```bash | ||
| curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": <mode>}' http://localhost:8081/mode | ||
| ``` | ||
| ==== | ||
|
|
||
| Replace the `<mode>` placeholder with the desired mode: | ||
|
|
||
| - `READONLY` | ||
| - `READWRITE` | ||
| - `IMPORT` | ||
|
|
||
| === Get mode for a subject | ||
|
|
||
| To look up the mode for a specific subject: | ||
|
|
||
| [tabs] | ||
| ==== | ||
| rpk:: | ||
| + | ||
| -- | ||
| ```bash | ||
| rpk registry mode get <subject-name> | ||
| ``` | ||
| -- | ||
|
|
||
| Curl:: | ||
| + | ||
| ```bash | ||
| curl http://localhost:8081/mode/<subject>?defaultToGlobal=true | ||
| ``` | ||
| ==== | ||
|
|
||
| This request returns the mode that is enforced. If the subject is set to a specific mode (to override the global mode), it returns the override mode. Otherwise, it returns the global mode. | ||
|
|
||
| To retrieve the subject-level override if it exists, use: | ||
|
|
||
| ```bash | ||
| curl http://localhost:8081/mode/<subject> | ||
| ``` | ||
|
|
||
| This request returns an error if there is no specific mode set for the subject. | ||
|
|
||
| === Set mode for a subject | ||
|
|
||
| [tabs] | ||
| ==== | ||
| rpk:: | ||
| + | ||
| -- | ||
| ```bash | ||
| rpk registry mode set <subject-name> --mode READONLY | ||
| ``` | ||
| -- | ||
|
|
||
| Curl:: | ||
| + | ||
| ```bash | ||
| curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "READONLY"}' http://localhost:8081/mode/<subject> | ||
| ``` | ||
| ==== | ||
|
|
||
| === Use READONLY mode for disaster recovery | ||
|
|
||
| A read-only Schema Registry does not accept direct writes. An active production cluster can replicate schemas to a read-only Schema Registry to keep it in sync, for example using Redpanda's https://github.com/redpanda-data/schema-migration/[Schema Migration tool^]. Users in the disaster recovery (DR) site cannot update schemas directly, so the DR cluster has an exact replica of the schemas in production. In a failover due to a disaster or outage, you can set Schema Registry to read-write mode, taking over for the failed cluster and ensuring availability. | ||
|
There was a problem hiding this comment. Maybe we rather want to recommend Redpanda Migrator here @pgellert @hcoyote ? https://docs.redpanda.com/redpanda-connect/cookbooks/redpanda_migrator/
Contributor
There was a problem hiding this comment. @hcoyote do we have any good customer examples of this by any chance? I don't know how well-maintained https://github.com/redpanda-data/schema-migration/ is at the moment, that's owned by CS as far as I know. The problem with linking to RPCN migrator here is that its schema_registry output requires the target cluster to be in IMPORT mode. So this described example of the target cluster being in READONLY mode only works if the replication is done as a replication of the topic, rather than a continuous schema registry import. We could still link to RPCN (https://docs.redpanda.com/redpanda-connect/cookbooks/redpanda_migrator/), but it might be good to have a blog post / docs page about how this is set up in practice if we have some good examples. I think DR will also require the target cluster's SR to be in READONLY mode. So maybe it would be even better to link to some Redpanda DR docs here? |
||
|
|
||
| === Use IMPORT mode for migration | ||
|
|
||
| Set the target Schema Registry to import mode to: | ||
|
|
||
| - Bypass compatibility checks when registering schemas. | ||
| - Specify a specific schema ID and version for the registered schema, so you can retain the same IDs and version from the original Schema Registry and keep topic data associated with the correct schema. | ||
|
|
||
| To enable import mode, you must have: | ||
|
|
||
| * Either superuser access, or a Schema Registry ACL with the `alter_configs` operation on the `registry` resource. See xref:manage:schema-reg/schema-reg-authorization.adoc#enable-schema-registry-authorization[Enable Schema Registry Authorization] to learn how to enable schema registry authorization for your cluster. | ||
| * An empty registry or subject. That is, either no schemas have ever been registered, or you must <<hard-delete-a-schema,hard-delete>> all schemas that were registered. | ||
| + | ||
| To bypass the check for an empty registry when setting the global mode to import: | ||
| + | ||
| [tabs] | ||
| ==== | ||
| rpk:: | ||
| + | ||
| -- | ||
| ```bash | ||
| rpk registry mode set --mode IMPORT --global --force | ||
| ``` | ||
| -- | ||
|
|
||
| Curl:: | ||
| + | ||
| ```bash | ||
| curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "IMPORT"}' http://localhost:8081/mode?force=true | ||
| ``` | ||
| ==== | ||
|
|
||
| Use import mode to register a schema with a specific ID and version: | ||
|
|
||
| [tabs] | ||
| ==== | ||
| rpk:: | ||
| + | ||
| -- | ||
| ```bash | ||
| rpk registry schema create <subject-name> --schema order.proto --id 1 --schema-version 4 | ||
kbatuigas marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ``` | ||
| -- | ||
|
|
||
| Curl:: | ||
| + | ||
| ```bash | ||
| curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "syntax = \"proto3\";\nmessage Order {\n string id = 1;\n}", "schemaType": "PROTOBUF", "id": 1, "version": 4}' http://localhost:8081/subjects/<subject-name>/versions | ||
| ``` | ||
| ==== | ||
|
|
||
| == Retrieve serialized schemas | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.