You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: This article provides information on using schemas in a schema registry when publishing or consuming events from Azure Event Hubs.
4
4
ms.topic: conceptual
5
5
ms.date: 04/26/2023
@@ -11,9 +11,9 @@ ms.author: spelluru
11
11
12
12
Client-side schema enforcement ensures that data is validated on the client side. The producer application sends the data and the consumer application receives it. That data is validated against schemas defined in the schema registry on the client side rather than the broker/server side.
13
13
14
-
This flow is illustrated as shown:
14
+
This diagram illustrates the flow:
15
15
16
-
:::image type="content" source="./media/schema-registry-overview/information-flow.svg" alt-text="A diagram outlines the schema registry information flow." border="false":::
16
+
:::image type="content" source="./media/schema-registry-overview/information-flow.svg" alt-text="Diagram that shows the schema registry information flow." border="false":::
17
17
18
18
> [!NOTE]
19
19
> The diagram showcases the information flow when event producers and consumers use a schema registry with the Kafka protocol and Avro schema. Other protocols and schema formats work in a similar way.
@@ -25,7 +25,7 @@ This flow is illustrated as shown:
25
25
26
26
1. The serializer looks for the schema in the schema registry to serialize event data. If it finds the schema, then the corresponding schema ID is returned. You can configure the producer application to automatically register the schema with the schema registry if it doesn't exist.
27
27
28
-
1. The serializer prepends the schema ID to the serialized data that is published to the Event Hubs.
28
+
1. The serializer prepends the schema ID to the serialized data that's published to the event hub.
Copy file name to clipboardExpand all lines: articles/event-hubs/schema-registry-concepts.md
+33-35Lines changed: 33 additions & 35 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,37 +7,37 @@ author: spelluru
7
7
ms.author: spelluru
8
8
---
9
9
10
-
# Schema Registry in Azure Event Hubs
10
+
# Schema registry concepts in Azure Event Hubs
11
11
12
12
Schema registries in Azure Event Hubs have many benefits. They help maintain data consistency, simplify schema evolution, enhance interoperability, and reduce development effort in loosely coupled and event-streaming workflows. Large distributed organizations that employ a centralized repository for schemas can use schema registries to achieve highly reliable data processing and governance with little operational overhead.
13
13
14
-
Azure Event Hubs schema registries fulfill many roles in schema-driven, event-streaming scenarios:
14
+
Azure Event Hubs schema registries fulfill many roles in schema-driven event-streaming scenarios:
15
15
16
-
* They provide a repository where multiple schemas can be registered, managed, and evolved
17
-
* They manage schema evolution with multiple compatibility rules
18
-
* They perform data validation for all schematized data
19
-
* They provide client-side libraries (serializers and deserializers) for producers and consumers
20
-
* They improve network throughput efficiency by passing the schema ID instead of the schema definition for every payload
16
+
* They provide a repository where multiple schemas can be registered, managed, and evolved.
17
+
* They manage schema evolution with multiple compatibility rules.
18
+
* They perform data validation for all schematized data.
19
+
* They provide client-side libraries (serializers and deserializers) for producers and consumers.
20
+
* They improve network throughput efficiency by passing the schema ID instead of the schema definition for every payload.
21
21
22
22
Azure Event Hubs schema registries are supported on Standard, Premium, and Dedicated tiers.
23
23
24
24
## Schema registry components
25
25
26
-
A schema registry lives in the context of the Event Hubs namespace, but can also be used with other message or events brokers, including Azure messaging services. It comprises multiple schema groups, which act as a logical grouping of schemas and can be managed independent of other schema groups.
26
+
A schema registry lives in the context of the Event Hubs namespace but can also be used with other message or event brokers, including Azure messaging services. It comprises multiple schema groups, which act as a logical grouping of schemas and can be managed independently of other schema groups.
27
27
28
-
:::image type="content" source="./media/schema-registry-overview/elements.png" alt-text="A diagram shows the components of a schema registry in Azure Event Hubs." border="false":::
28
+
:::image type="content" source="./media/schema-registry-overview/elements.png" alt-text="Diagram that shows the components of a schema registry in Azure Event Hubs." border="false":::
29
29
30
30
### Schemas
31
31
32
-
In any loosely coupled system, multiple applications communicate, primarily through data. Schemas define the structure of the data in a declarative way. As a result, the contract between producer and consumer applications is well-defined, ensuring reliable processing at scale.
32
+
In any loosely coupled system, multiple applications communicate, primarily through data. Schemas define the structure of the data in a declarative way. As a result, the contract between producer and consumer applications is welldefined, ensuring reliable processing at scale.
33
33
34
34
A schema definition includes:
35
35
36
-
* Fields: Individual data elements like name, book title, or address
37
-
* Data types: The type of data that can be stored, like string, date-time, or array
38
-
* Structure: How the fields are organized, like nested structures or arrays
36
+
* Fields: Individual data elements like name, book title, or address.
37
+
* Data types: The type of data that can be stored, like string, date-time, or array.
38
+
* Structure: How the fields are organized, like nested structures or arrays.
39
39
40
-
Schemas define the contract between producers and consumers. A schema defined in an Event Hubs schema registry helps manage the contract outside of event data, which removes the payload overhead.
40
+
Schemas define the contract between producers and consumers. A schema defined in an Event Hubs schema registry helps manage the contract outside event data, which removes the payload overhead.
41
41
42
42
#### Schema formats
43
43
@@ -54,31 +54,29 @@ To learn more about using Avro schema format with an Event Hubs schema registry,
54
54
55
55
##### JSON schema
56
56
57
-
[JSON (JavaScript Object Notation) schema](https://json-schema.org/) is a standardized way of defining the structure and data types of the events. JSON schema enables the confident and reliable use of the JSON data format in event streaming.
57
+
A [JSON (JavaScript Object Notation) schema](https://json-schema.org/) is a standardized way of defining the structure and data types of the events. A JSON schema enables the confident and reliable use of the JSON data format in event streaming.
58
58
59
-
To learn more about using the JSON schema format with an Event Hubs schema registry, see:
60
-
61
-
*[How to use a schema registry with Kafka and JSON schema](schema-registry-json-schema-kafka.md)
59
+
To learn more about using the JSON schema format with an Event Hubs schema registry, see [How to use a schema registry with Kafka and JSON schema](schema-registry-json-schema-kafka.md).
62
60
63
61
##### Protocol Buffers
64
62
65
-
[Protocol Buffers (Protobuf)](https://protobuf.dev/) is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. It's used for efficiently defining data structures and serializing them into a compact, binary format.
63
+
[Protocol Buffers (Protobuf)](https://protobuf.dev/) is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. It's used for efficiently defining data structures and serializing them into a compact binary format.
66
64
67
65
### Schema groups
68
66
69
67
Schema groups are logical groups of similar schemas that are organized according to your business criteria. A schema group holds:
70
68
71
-
* Multiple schema definitions
72
-
* Multiple versions of a specific schema
73
-
* Metadata regarding the schema type and compatibility for all schemas in the group
69
+
* Multiple schema definitions.
70
+
* Multiple versions of a specific schema.
71
+
* Metadata regarding the schema type and compatibility for all schemas in the group.
74
72
75
-
You can think of a schema group as a subset of the schema registry that aligns with a particular application or organizational unit, with a separate authorization model. This extra security boundary ensures that metadata and trade secrets aren't leaked in the shared services model. It also allows application owners to manage schemas independent of other applications that share the same namespace.
73
+
You can think of a schema group as a subset of the schema registry that aligns with a particular application or organizational unit, with a separate authorization model. This extra security boundary helps ensure that metadata and trade secrets aren't leaked in the shared services model. It also allows application owners to manage schemas independently of other applications that share the same namespace.
76
74
77
75
## Schema evolution
78
76
79
-
Schemas need to evolve with the business requirement of producers and consumers. Azure schema registries support schema evolution by introducing compatibility modes at the schema group level. When you create a schema group, you can specify the compatibility mode of the schemas that you include in that schema group. When you update a schema, the change needs to comply with the assigned compatibility mode in order for it to create a new version of the schema.
77
+
Schemas need to evolve with the business requirement of producers and consumers. Azure schema registries support schema evolution by introducing compatibility modes at the schema group level. When you create a schema group, you can specify the compatibility mode of the schemas that you include in that schema group. When you update a schema, the change needs to comply with the assigned compatibility mode so that it can create a new version of the schema.
80
78
81
-
Schema evolution is only supported for Avro schema format only.
79
+
Schema evolution is supported for Avro schema format only.
82
80
83
81
Azure Event Hubs schema registries are supported in the following compatibility modes.
84
82
@@ -102,15 +100,15 @@ When the ``None`` compatibility mode is used, the schema registry doesn't do any
102
100
103
101
## Client SDKs
104
102
105
-
You can use one of the following libraries to include an Avro serializer. You can use Avro serializers to serialize and deserialize payloads containing schema registry schema identifiers and Avro-encoded data:
103
+
You can use one of the following libraries to include an Avro serializer. You can use Avro serializers to serialize and deserialize payloads that contain schema registry schema identifiers and Avro-encoded data:
***Apache Kafka**: Run [Kafka-integrated Avro](https://github.com/Azure/azure-schema-registry-for-kafka/) serializers and deserializers backed by Azure Schema Registry. The Java client's Apache Kafka® client serializer for Azure schema registries can be used in any Apache Kafka scenario and with any Apache Kafka-based deployment or cloud service.
112
-
***Azure CLI**: For an example of adding a schema to a schema group by using CLI, see [Adding a schema to a schema group using CLI](https://github.com/Azure/azure-event-hubs/tree/master/samples/Management/CLI/AddschematoSchemaGroups).
113
-
***PowerShell**: For an example of adding a schema to a schema group by using PowerShell, see [Adding a schema to a schema group using PowerShell](https://github.com/Azure/azure-event-hubs/tree/master/samples/Management/PowerShell/AddingSchematoSchemagroups).
109
+
***Apache Kafka**: Run [Kafka-integrated Avro](https://github.com/Azure/azure-schema-registry-for-kafka/) serializers and deserializers backed by Azure schema registries. The Java client's Apache Kafka® client serializer for Azure schema registries can be used in any Apache Kafka scenario and with any Apache Kafka-based deployment or cloud service.
110
+
***Azure CLI**: For an example of adding a schema to a schema group by using the Azure CLI, see [Adding a schema to a schema group by using the Azure CLI](https://github.com/Azure/azure-event-hubs/tree/master/samples/Management/CLI/AddschematoSchemaGroups).
111
+
***PowerShell**: For an example of adding a schema to a schema group by using PowerShell, see [Adding a schema to a schema group by using PowerShell](https://github.com/Azure/azure-event-hubs/tree/master/samples/Management/PowerShell/AddingSchematoSchemagroups).
114
112
115
113
## Limits
116
114
@@ -127,17 +125,17 @@ To access a schema registry programmatically, follow these steps:
127
125
| ---- | ----------- |
128
126
| Owner | Read, write, and delete schema registry groups and schemas |
129
127
| Contributor | Read, write, and delete schema registry groups and schemas |
130
-
|[Schema registry reader](../role-based-access-control/built-in-roles.md#schema-registry-reader-preview)| Read and list schema registry groups and schemas |
131
-
|[Schema registry contributor](../role-based-access-control/built-in-roles.md#schema-registry-reader-preview)| Read, write, and delete schema registry groups and schemas |
128
+
|[Schema Registry Reader](../role-based-access-control/built-in-roles.md#schema-registry-reader-preview)| Read and list schema registry groups and schemas |
129
+
|[Schema Registry Contributor](../role-based-access-control/built-in-roles.md#schema-registry-reader-preview)| Read, write, and delete schema registry groups and schemas |
132
130
133
-
To learn how to create and register an application using the Azure portal, see [Register an application with Microsoft Entra ID](../active-directory/develop/quickstart-register-app.md). You need the client ID (application ID), tenant ID, and the secret to use in the code.
131
+
To learn how to create and register an application by using the Azure portal, see [Register an application with Microsoft Entra ID](../active-directory/develop/quickstart-register-app.md). You need the client ID (application ID), the tenant ID, and the secret to use in the code.
134
132
135
-
## Next steps
133
+
## Related content
136
134
137
-
* To learn how to create a schema registry by using the Azure portal, see [Create an Event Hubs schema registry using the Azure portal](create-schema-registry.md).
135
+
* To learn how to create a schema registry by using the Azure portal, see [Create an Event Hubs schema registry by using the Azure portal](create-schema-registry.md).
138
136
* See the following **schema registry Avro client library** samples.
0 commit comments