[25.3] Import mode for Schema Registry #1391
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
|
|
||
| === Use READONLY mode for disaster recovery | ||
|
|
||
| A read-only Schema Registry does not accept direct writes. An active production cluster can replicate schemas to a read-only Schema Registry to keep it in sync, for example using Redpanda's https://github.com/redpanda-data/schema-migration/[Schema Migration tool^]. Users in the disaster recovery (DR) site cannot update schemas directly, so the DR cluster has an exact replica of the schemas in production. In a failover due to a disaster or outage, you can set Schema Registry to read-write mode, taking over for the failed cluster and ensuring availability. |
There was a problem hiding this comment.
Maybe we rather want to recommend Redpanda Migrator here @pgellert @hcoyote ?
https://docs.redpanda.com/redpanda-connect/cookbooks/redpanda_migrator/
There was a problem hiding this comment.
@hcoyote do we have any good customer examples of this by any chance?
I don't know how well-maintained https://github.com/redpanda-data/schema-migration/ is at the moment, that's owned by CS as far as I know. The problem with linking to RPCN migrator here is that its schema_registry output requires the target cluster to be in IMPORT mode. So this described example of the target cluster being in READONLY mode only works if the replication is done as a replication of the topic, rather than a continuous schema registry import.
We could still link to RPCN (https://docs.redpanda.com/redpanda-connect/cookbooks/redpanda_migrator/), but it might be good to have a blog post / docs page about how this is set up in practice if we have some good examples.
I think DR will also require the target cluster's SR to be in READONLY mode. So maybe it would be even better to link to some Redpanda DR docs here?
| - Bypass compatibility checks when registering schemas. | ||
| - Specify a specific schema ID and version for the registered schema, so you can retain the same IDs and version from the original Schema Registry and keep topic data associated with the correct schema. | ||
|
|
||
| To enable import mode, you need either superuser access, or a Schema Reigstry ACL with the `alter_configs` operation on the `registry` resource. See xref:manage:schema-reg/schema-reg-authorization.adoc#enable-schema-registry-authorization[Enable Schema Registry Authorization] to learn how to enable schema registry authorization for your cluster. |
| ==== | ||
| In Redpanda versions before 25.3, you could specify a schema ID or version when registering a schema in read-write mode. | ||
| Starting with 25.3, read-write mode returns an error when you try to register a schema with a specific ID or version. If you have custom scripts that rely on the ability to specify an ID or version with Redpanda 25.2 and earlier, you must do either of the following: |
There was a problem hiding this comment.
@kbatuigas this also needs to be included in the 'upgrade guide' noting incompatible changes in 25.3.
There was a problem hiding this comment.
cc @Feediver1 @Jeffail @david-yu @hcoyote we need to ensure Migrator adapts to this as cleanly as possible (Migrator knows when to start using IMPORT mode so imports don't just start failing in readwrite mode when customers with active Migrator pipelines upgrade).
| * Change the Schema Registry or the subject to import mode. | ||
| ==== | ||
|
|
||
| To force enable import mode: |
There was a problem hiding this comment.
@kbatuigas the meaning of force isn't explained here. What's the different between force and not?
There was a problem hiding this comment.
+1, @kbatuigas I think you could just include this explanation here: https://redpandadata.atlassian.net/wiki/spaces/CORE/pages/1345224705/Schema+Registry+IMPORT+mode
| - xref:reference:properties/cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] | ||
| - xref:reference:properties/cluster-properties.adoc#schema_registry_enable_authorization[`schema_registry_enable_authorization`] | ||
| - xref:reference:properties/cluster-properties.adoc#topic_label_aggregation_limit[`topic_label_aggregation_limit`] | ||
| Redpanda Schema Registry now supports an import mode that allows you to import existing schemas and retain their current IDs and version numbers. Import mode is useful when migrating from another schema registry. See xref:manage:schema-reg/schema-reg-api.adoc#set-schema-registry-mode[Use the Schema Registry API] for more information. |
There was a problem hiding this comment.
I think we should call out here the breaking change that previously READWRITE mode allowed specifying the id + version of the schema when creating the schema, but now READWRITE mode no longer allows that. Maybe a sentence at the end:
"Going forward, IMPORT mode must be used when importing schemas. READWRITE mode no longer allows specifying a schema id and schema version when creating a schema."
|
|
||
| === Use READONLY mode for disaster recovery | ||
|
|
||
| A read-only Schema Registry does not accept direct writes. An active production cluster can replicate schemas to a read-only Schema Registry to keep it in sync, for example using Redpanda's https://github.com/redpanda-data/schema-migration/[Schema Migration tool^]. Users in the disaster recovery (DR) site cannot update schemas directly, so the DR cluster has an exact replica of the schemas in production. In a failover due to a disaster or outage, you can set Schema Registry to read-write mode, taking over for the failed cluster and ensuring availability. |
There was a problem hiding this comment.
@hcoyote do we have any good customer examples of this by any chance?
I don't know how well-maintained https://github.com/redpanda-data/schema-migration/ is at the moment, that's owned by CS as far as I know. The problem with linking to RPCN migrator here is that its schema_registry output requires the target cluster to be in IMPORT mode. So this described example of the target cluster being in READONLY mode only works if the replication is done as a replication of the topic, rather than a continuous schema registry import.
We could still link to RPCN (https://docs.redpanda.com/redpanda-connect/cookbooks/redpanda_migrator/), but it might be good to have a blog post / docs page about how this is set up in practice if we have some good examples.
I think DR will also require the target cluster's SR to be in READONLY mode. So maybe it would be even better to link to some Redpanda DR docs here?
| * Change the Schema Registry or the subject to import mode. | ||
| ==== | ||
|
|
||
| To force enable import mode: |
There was a problem hiding this comment.
+1, @kbatuigas I think you could just include this explanation here: https://redpandadata.atlassian.net/wiki/spaces/CORE/pages/1345224705/Schema+Registry+IMPORT+mode
kbatuigas
deleted the
DOC-1194-document-feature-import-mode-for-schema-registry
branch
Description
Related: API reference PR
This pull request updates the Schema Registry API documentation to clarify and expand the information about supported modes, including new instructions and use cases for disaster recovery and migration. The changes introduce the new
IMPORTmode, reorganize documentation for clarity, and add examples for bothrpkandcurlcommands.Schema Registry Modes and Usage
IMPORTmode, explaining its use for migration scenarios, including bypassing compatibility checks and specifying schema IDs and versions. [1] [2]READWRITE,READONLY, andIMPORTmodes, and reorganized the documentation to provide clear guidance on when to use each mode.Command Examples and Instructions
rpkandcurlcommands throughout the documentation, making it easier for users to find and use the correct commands for their setup. [1] [2]Disaster Recovery and Migration Guidance
Resolves https://redpandadata.atlassian.net/browse/
Review deadline: 16 Oct
Page previews
Use the Schema Registry API > Set Schema Registry mode
Checks