You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/event-hubs/schema-registry-concepts.md
+35-13Lines changed: 35 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,41 +8,63 @@ ms.author: spelluru
8
8
---
9
9
10
10
# Schema Registry in Azure Event Hubs
11
-
Schema Registry in Azure Event Hubs provides you with a repository to use and manage schemas in schema-driven event streaming scenarios.
12
11
13
-
> [!NOTE]
14
-
> Schema Registry is not supported on Basic tier.
12
+
Schema Registry in Azure Event Hubs fulfills multiple roles in schema-driven event streaming scenarios -
13
+
* Provides a repository where multiple schemas can be registered, managed, and evolved.
14
+
* Performs data validation for all schematized data.
15
+
* Provides client-side libraries (serializers and deserializers) for producers and consumers.
16
+
17
+
> [!NOTE]
18
+
> Schema Registry is supported on Standard, Premium and Dedicated tiers.
19
+
>
15
20
16
-
## Schema Registry components
21
+
## Schema Registry components
17
22
18
-
An Event Hubs namespace can host schema groups alongside event hubs (or Kafka topics). It hosts a schema registry and can have multiple schema groups. In spite of being hosted in Azure Event Hubs, the schema registry can be used universally with all Azure messaging services and any other message or events broker. Each of these schema groups is a separately securable repository for a set of schemas. Groups can be aligned with a particular application or an organizational unit.
23
+
The Schema Registry lives in the context of the Event Hubs namespace, but it can be used with all Azure messaging service or other message or events broker. It comprises multiple schema groups which acts as a logical grouping of schemas and can be managed independent of other schema groups.
19
24
20
25
:::image type="content" source="./media/schema-registry-overview/elements.png" alt-text="Diagram that shows the components of Schema Registry in Azure Event Hubs." border="false":::
21
26
22
-
### Schema groups
23
-
Schema group is a logical group of similar schemas based on your business criteria. A schema group can hold multiple versions of a schema. The compatibility enforcement setting on a schema group can help ensure that newer schema versions are backwards compatible.
27
+
### Schemas
24
28
25
-
The security boundary imposed by the grouping mechanism help ensures that trade secrets don't inadvertently leak through metadata in situations where the namespace is shared among multiple partners. It also allows for application owners to manage schemas independent of other applications that share the same namespace.
29
+
In any loosely coupled system, there are multiple applications communicating with each other, primarily through data. Schemas act as a declarative way to define the structure of the data so that the contract between these producer and consumer applications is well defined, ensuring reliable processing at scale.
26
30
27
-
### Schemas
28
-
Schemas define the contract between producers and consumers. A schema defined in an Event Hubs schema registry helps manage the contract outside of event data, thus removing the payload overhead. A schema has a name, type (example: record, array, and so on.), compatibility mode (none, forward, backward, full), and serialization type (both Avro and JSON). You can create multiple versions of a schema and retrieve and use a specific version of a schema.
31
+
A schema definition includes -
32
+
* Fields - name of the individual data elements (i.e. first/last name, book title, address).
33
+
* Data types - the kind of data that can be stored in each field (e.g. string, date-time, array).
34
+
* Structure - the organization of the different fields (i.e. nested structures or arrays).
35
+
36
+
Schemas define the contract between producers and consumers. A schema defined in an Event Hubs schema registry helps manage the contract outside of event data, thus removing the payload overhead.
29
37
30
-
### Schema formats
38
+
####Schema formats
31
39
Schema formats are used to determine the manner in which a schema is structured and defined, with each format outlining specific guidelines and syntax for defining the structure of the events that will be used for event streaming.
32
40
33
-
#### Avro schema
41
+
#####Avro schema
34
42
[Avro](https://avro.apache.org/) is a popular data serialization system that uses a compact binary format and provides schema evolution capabilities.
35
43
36
44
To learn more about using Avro schema format with Event Hubs Schema Registry, see:
37
45
-[How to use schema registry with Kafka and Avro](schema-registry-kafka-java-send-receive-quickstart.md)
38
46
-[How to use Schema registry with Event Hubs .NET SDK (AMQP) and Avro.](schema-registry-dotnet-send-receive-quickstart.md)
39
47
40
-
#### JSON Schema
48
+
#####JSON Schema
41
49
[JSON Schema](https://json-schema.org/) is a standardized way of defining the structure and data types of the events. JSON Schema enables the confident and reliable use of the JSON data format in event streaming.
42
50
43
51
To learn more about using JSON schema format with Event Hubs Schema Registry, see:
44
52
-[How to use schema registry with Kafka and JSON Schema](schema-registry-json-schema-kafka.md)
45
53
54
+
##### Protobuf
55
+
56
+
TODO
57
+
58
+
59
+
### Schema groups
60
+
61
+
Schema groups are logical groups of similar schemas based on your business criteria. A schema group holds
62
+
* multiple schema definitons,
63
+
* multiple versions of a specific schema, and
64
+
* metadata regarding the schema type and compatibility for all schemas in the group.
65
+
66
+
A schema groups can be thought of as a subset of the schema registry, aligned with a particular application or organizational unit, with a separate authorization model. This additional security boundary ensures that in the shared services model, metadata and trade secrets are not leaked. It also allows for application owners to manage schemas independent of other applications that share the same namespace.
67
+
46
68
## Schema evolution
47
69
Schemas need to evolve with the business requirement of producers and consumers. Azure Schema Registry supports schema evolution by introducing compatibility modes at the schema group level. When you create a schema group, you can specify the compatibility mode of the schemas that you include in that schema group. When you update a schema, the change should comply with the assigned compatibility mode and then only it creates a new version of the schema.
0 commit comments