Merge pull request #292728 from axisc/schema-registry-docs-refresh

Schema registry docs refresh

Author: Jill Grant

articles/event-hubs/schema-registry-client-side-enforcement.md

@@ -7,22 +7,32 @@ author: spelluru
 ms.author: spelluru
 ---
 
-# Client-side schema enforcement 
-The information flow when you use schema registry is the same for all protocols that you use to publish or consume events from Azure Event Hubs. 
+# Client-side schema enforcement
 
-The following diagram shows how the information flows when event producers and consumers use Schema Registry with the **Kafka** protocol using **Avro** serialization.  
+Client-side schema enforcement ensures that the data sent by the producer application and received by the consumer application is validated against the schemas defined in the Schema Registry on the client side itself (that is, rather than on the broker/server side).
+
+This flow is illustrated as shown - 
 
 :::image type="content" source="./media/schema-registry-overview/information-flow.svg" alt-text="Image showing the Schema Registry information flow." border="false":::
 
+> [!NOTE]
+> While the diagram showcases the information flow when event producers and consumers use Schema Registry with the **Kafka** protocol and **Avro** schema, it doesn't really change for other protocols and schema formats.
+>
+
 ### Producer  
 
 1. Kafka producer application uses `KafkaAvroSerializer` to serialize event data using the specified schema. Producer application provides details of the schema registry endpoint and other optional parameters that are required for schema validation. 
-1. The serializer looks for the schema in the schema registry to serialize event data. If it finds the schema, then the corresponding schema ID is returned. You can configure the producer application to auto register the schema with the schema registry if it doesn't exist. 
-1. Then the serializer prepends the schema ID to the serialized data that is published to the Event Hubs. 
+
+2. The serializer looks for the schema in the schema registry to serialize event data. If it finds the schema, then the corresponding schema ID is returned. You can configure the producer application to auto register the schema with the schema registry if it doesn't exist.
+
+3. Then the serializer prepends the schema ID to the serialized data that is published to the Event Hubs. 
 
 ### Consumer 
 
 1. Kafka consumer application uses `KafkaAvroDeserializer` to deserialize data that it receives from the event hub.
-1. The deserializer uses the schema ID (prepended by the producer) to retrieve schema from the schema registry.
-1. The deserializer uses the schema to deserialize event data that it receives from the event hub. 
-1. The schema registry client uses caching to prevent redundant schema registry lookups in the future.  
+
+2. The deserializer uses the schema ID (prepended by the producer) to retrieve schema from the schema registry.
+
+3. The deserializer uses the schema to deserialize event data that it receives from the event hub. 
+
+4. The schema registry client uses caching to prevent redundant schema registry lookups in the future.  

articles/event-hubs/schema-registry-concepts.md

@@ -8,41 +8,66 @@ ms.author: spelluru
 ---
 
 #  Schema Registry in Azure Event Hubs
-Schema Registry in Azure Event Hubs provides you with a repository to use and manage schemas in schema-driven event streaming scenarios. 
 
- > [!NOTE]
-> Schema Registry is not supported on Basic tier.
+Schema Registry is crucial in loosely coupled and event streaming workflows for maintaining data consistency, simplifying schema evolution, enhancing interoperability, and reducing development effort. It ensures highly reliable data processing and governance with little operational overhead in large distributed organizations with a centralized repository for schemas.
 
-## Schema Registry components 
+Schema Registry in Azure Event Hubs fulfills multiple roles in schema-driven event streaming scenarios -
+   * Provides a repository where multiple schemas can be registered, managed, and evolved.
+   * Managed schema evolution with multiple compatibility rules.
+   * Performs data validation for all schematized data.
+   * Provides client-side libraries (serializers and deserializers) for producers and consumers.
+   * Improves network throughput efficiency by passing schema ID instead of the schema definition for every payload.
 
-An Event Hubs namespace can host schema groups alongside event hubs (or Kafka topics). It hosts a schema registry and can have multiple schema groups. In spite of being hosted in Azure Event Hubs, the schema registry can be used universally with all Azure messaging services and any other message or events broker. Each of these schema groups is a separately securable repository for a set of schemas. Groups can be aligned with a particular application or an organizational unit. 
+> [!NOTE]
+> Schema Registry is supported on Standard, Premium, and Dedicated tiers.
+>
 
-:::image type="content" source="./media/schema-registry-overview/elements.png" alt-text="Diagram that shows the components of Schema Registry in Azure Event Hubs." border="false":::
+## Schema Registry components
 
-### Schema groups
-Schema group is a logical group of similar schemas based on your business criteria. A schema group can hold multiple versions of a schema. The compatibility enforcement setting on a schema group can help ensure that newer schema versions are backwards compatible.
+The Schema Registry lives in the context of the Event Hubs namespace, but it can be used with all Azure messaging service or other message or events broker. It comprises multiple schema groups which act as a logical grouping of schemas and can be managed independent of other schema groups. 
 
-The security boundary imposed by the grouping mechanism help ensures that trade secrets don't inadvertently leak through metadata in situations where the namespace is shared among multiple partners. It also allows for application owners to manage schemas independent of other applications that share the same namespace.
+:::image type="content" source="./media/schema-registry-overview/elements.png" alt-text="Diagram that shows the components of Schema Registry in Azure Event Hubs." border="false":::
 
 ### Schemas
-Schemas define the contract between producers and consumers. A schema defined in an Event Hubs schema registry helps manage the contract outside of event data, thus removing the payload overhead. A schema has a name, type (example: record, array, and so on.), compatibility mode (none, forward, backward, full), and serialization type (both Avro and JSON). You can create multiple versions of a schema and retrieve and use a specific version of a schema. 
 
-### Schema formats 
+In any loosely coupled system, there are multiple applications communicating with each other, primarily through data. Schemas act as a declarative way to define the structure of the data so that the contract between these producer and consumer applications is well defined, ensuring reliable processing at scale.
+
+A schema definition includes -
+   * Fields - name of the individual data elements (that is, first/last name, book title, address).
+   * Data types - the kind of data that can be stored in each field (for example, string, date-time, array).
+   * Structure - the organization of the different fields (that is, nested structures or arrays).
+
+Schemas define the contract between producers and consumers. A schema defined in an Event Hubs schema registry helps manage the contract outside of event data, thus removing the payload overhead.
+
+#### Schema formats 
 Schema formats are used to determine the manner in which a schema is structured and defined, with each format outlining specific guidelines and syntax for defining the structure of the events that will be used for event streaming.
 
-#### Avro schema 
+##### Avro schema 
 [Avro](https://avro.apache.org/) is a popular data serialization system that uses a compact binary format and provides schema evolution capabilities. 
 
 To learn more about using Avro schema format with Event Hubs Schema Registry, see:  
 - [How to use schema registry with Kafka and Avro](schema-registry-kafka-java-send-receive-quickstart.md)
 - [How to use Schema registry with Event Hubs .NET SDK (AMQP) and Avro.](schema-registry-dotnet-send-receive-quickstart.md)
 
-#### JSON Schema
+##### JSON Schema
 [JSON Schema](https://json-schema.org/) is a standardized way of defining the structure and data types of the events. JSON Schema enables the confident and reliable use of the JSON data format in event streaming. 
 
 To learn more about using JSON schema format with Event Hubs Schema Registry, see:  
 - [How to use schema registry with Kafka and JSON Schema](schema-registry-json-schema-kafka.md)
 
+##### Protobuf
+
+[Protocol Buffers](https://protobuf.dev/) is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. It's used for efficiently defining data structures and serializing them into a compact binary format.
+
+### Schema groups
+
+Schema groups are logical groups of similar schemas based on your business criteria. A schema group holds 
+   * multiple schema definition, 
+   * multiple versions of a specific schema, and 
+   * metadata regarding the schema type and compatibility for all schemas in the group.
+
+A schema groups can be thought of as a subset of the schema registry, aligned with a particular application or organizational unit, with a separate authorization model. This extra security boundary ensures that in the shared services model, metadata, and trade secrets aren't leaked. It also allows for application owners to manage schemas independent of other applications that share the same namespace.
+
 ## Schema evolution 
 Schemas need to evolve with the business requirement of producers and consumers. Azure Schema Registry supports schema evolution by introducing compatibility modes at the schema group level. When you create a schema group, you can specify the compatibility mode of the schemas that you include in that schema group. When you update a schema, the change should  comply with the assigned compatibility mode and then only it creates a new version of the schema. 
 
@@ -85,7 +110,7 @@ For limits (for example: number of schema groups in a namespace) of Event Hubs,
 To access a schema registry programmatically, follow these steps:
 
 1. [Register your application in Microsoft Entra ID](../active-directory/develop/quickstart-register-app.md)
-1. Add the security principal of the application to one of the following Azure role-based access control (Azure RBAC) roles at the **namespace** level. 
+1. Add the security principal of the application to one of the following Azure RBAC(role-based access control) roles at the **namespace** level. 
 
 | Role | Description | 
 | ---- | ----------- | 

articles/event-hubs/schema-registry-overview.md

@@ -8,23 +8,27 @@ ms.custom: references_regions
 ---
 
 # Azure Schema Registry in Event Hubs
-In many event streaming and messaging scenarios, the event or message payload contains structured data. Schema-driven formats such as [Apache Avro](https://avro.apache.org/) are often used to serialize or deserialize such structured data. 
 
-An event producer uses a schema to serialize event payload and publish it to an event broker such as Event Hubs. Event consumers read event payload from the broker and deserialize it using the same schema. So, both producers and consumers can validate the integrity of the data with the same schema. 
+Event streaming and messaging scenarios often deal with structured data in the event or message payload. However, the structured data is of little value to the event broker, which only deals with bytes. Schema-driven formats such as [Apache Avro](https://avro.apache.org/), [JSONSchema](https://json-schema.org/), or [Protobuf](https://protobuf.dev/) are often used to serialize or deserialize such structured data to/from binary.
+
+An event producer uses a schema definition to serialize event payload and publish it to an event broker such as Event Hubs. Event consumers read event payload from the broker and deserialize it using the same schema definition. 
+
+So, both producers and consumers can validate the integrity of the data with the same schema. 
 
 :::image type="content" source="./media/schema-registry-overview/schema-driven-ser-de.svg" alt-text="Image showing producers and consumers serializing and deserializing event payload using schemas from the Schema Registry. ":::
 
 ## What is Azure Schema Registry?
-**Azure Schema Registry** is a feature of Event Hubs, which provides a central repository for schemas for event-driven and messaging-centric applications. It provides the flexibility for your producer and consumer applications to **exchange data without having to manage and share the schema**. It also provides a simple governance framework for reusable schemas and defines relationship between schemas through a grouping construct (schema groups).
+**Azure Schema Registry** is a feature of Event Hubs, which provides a central repository for schemas for event-driven and messaging-centric applications. It provides the flexibility for your producer and consumer applications to **exchange data without having to manage and share the schema**. It also provides a simple governance framework for reusable schemas and defines relationship between schemas through a logical grouping construct (schema groups).
 
 :::image type="content" source="./media/schema-registry-overview/schema-registry.svg" alt-text="Image showing a producer and a consumer serializing and deserializing event payload using a schema from the Schema Registry." border="false":::
 
-With schema-driven serialization frameworks like Apache Avro, moving serialization metadata into shared schemas can also help with **reducing the per-message overhead**. It's because each message doesn't need to have the metadata (type information and field names) as it's the case with tagged formats such as JSON. 
+With schema-driven serialization frameworks like Apache Avro, JSONSchema and Protobuf, moving serialization metadata into shared schemas can also help with **reducing the per-message overhead**. It's because each message doesn't need to have the metadata (type information and field names) as it's the case with tagged formats such as JSON. 
 
 > [!NOTE]
-> The feature isn't available in the **basic** tier.
+> The feature is available in the **Standard**, **Premium**, and **Dedicated** tier.
+>
 
-Having schemas stored alongside the events and inside the eventing infrastructure ensures that the metadata that's required for serialization or deserialization is always in reach and schemas can't be misplaced. 
+Having schemas stored alongside the events and inside the eventing infrastructure ensures that the metadata required for serialization or deserialization is always in reach and schemas can't be misplaced. 
 
 ## Related content