Merge pull request #235915 from kasun04/main

Add JSON schema support and restructure SR docs

Author: Jill Grant

articles/event-hubs/TOC.yml

@@ -106,6 +106,12 @@
         href: event-hubs-dedicated-overview.md
       - name: Compare tiers
         href: compare-tiers.md
+    - name: Schema Registry 
+      items: 
+      - name: Schema Registry concepts
+        href: schema-registry-concepts.md   
+      - name: Client-side schema enforcement
+        href: schema-registry-client-side-enforcement.md
     - name: Event processing using SDKs
       items: 
       - name: Event processor host (legacy SDK)
@@ -232,6 +238,22 @@
           href: ../stream-analytics/no-code-filter-ingest-data-explorer.md?toc=/azure/event-hubs/toc.json
         - name: Process data using Azure Stream Analytics
           href: process-data-azure-stream-analytics.md
+    
+    - name: Use Schema Registry 
+      items:
+        - name: Client-side schema enforcement 
+          items:
+          - name: Using Avro schema 
+            items: 
+            - name: Avro schema with Apache Kafka applications
+              href: schema-registry-kafka-java-send-receive-quickstart.md
+            - name: Avro schema with Event Hubs SDK(AMQP)
+              href: schema-registry-dotnet-send-receive-quickstart.md
+          - name: Using JSON Schema  
+            items: 
+            - name: JSON Schema with Apache Kafka applications
+              href: schema-registry-json-schema-kafka.md
+
     - name: Integrate with Spring framework
       items:
         - name: Create a Spring Cloud Stream Binder app

articles/event-hubs/authenticate-application.md

@@ -21,7 +21,7 @@ Azure provides the following Azure built-in roles for authorizing access to Even
 - [Azure Event Hubs Data Sender](../role-based-access-control/built-in-roles.md#azure-event-hubs-data-sender): Use this role to give access to Event Hubs resources.
 - [Azure Event Hubs Data Receiver](../role-based-access-control/built-in-roles.md#azure-event-hubs-data-receiver): Use this role to give receiving access to Event Hubs resources.   
 
-For Schema Registry built-in roles, see [Schema Registry roles](schema-registry-overview.md#azure-role-based-access-control).
+For Schema Registry built-in roles, see [Schema Registry roles](schema-registry-concepts.md#azure-role-based-access-control).
 
 > [!IMPORTANT]
 > Our preview release supported adding Event Hubs data access privileges to Owner or Contributor role. However, data access privileges for Owner and Contributor role are no longer honored. If you are using the Owner or Contributor role, switch to using the Azure Event Hubs Data Owner role.

articles/event-hubs/authorize-access-azure-active-directory.md

@@ -34,7 +34,7 @@ Azure provides the following Azure built-in roles for authorizing access to Even
 | [Azure Event Hubs Data sender](../role-based-access-control/built-in-roles.md#azure-event-hubs-data-sender) | Use this role to give the send access to Event Hubs resources. |
 | [Azure Event Hubs Data receiver](../role-based-access-control/built-in-roles.md#azure-event-hubs-data-receiver) | Use this role to give the consuming/receiving access to Event Hubs resources. |
 
-For Schema Registry built-in roles, see [Schema Registry roles](schema-registry-overview.md#azure-role-based-access-control).
+For Schema Registry built-in roles, see [Schema Registry roles](schema-registry-concepts.md#azure-role-based-access-control).
 
 ## Resource scope 
 Before you assign an Azure role to a security principal, determine the scope of access that the security principal should have. Best practices dictate that it's always best to grant only the narrowest possible scope.

articles/event-hubs/create-schema-registry.md

@@ -14,7 +14,7 @@ This article shows you how to create a schema group with schemas in a schema reg
 
 > [!NOTE]
 > - The feature isn't available in the **basic** tier.
-> - Make sure that you are a member of one of these roles: **Owner**, **Contributor**, or **Schema Registry Contributor**. For details about role-based access control, see [Schema Registry overview](schema-registry-overview.md#azure-role-based-access-control).
+> - Make sure that you are a member of one of these roles: **Owner**, **Contributor**, or **Schema Registry Contributor**. For details about role-based access control, see [Schema Registry overview](schema-registry-concepts.md#azure-role-based-access-control).
 > - If the event hub is in a **virtual network**, you won't be able to create schemas in the Azure portal unless you access the portal from a VM in the same virtual network. 
 
 

articles/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs.md

@@ -3,11 +3,13 @@ title: 'Quickstart: Use Apache Kafka with Azure Event Hubs'
 description: 'This quickstart shows you how to stream data into and from Azure Event Hubs using the Apache Kafka protocol.'
 ms.topic: quickstart
 ms.date: 02/07/2023
+author: kasun04
+ms.author: kindrasiri
 ms.custom: mode-other, passwordless-java
 ---
 
 # Quickstart: Stream data with Azure Event Hubs and Apache Kafka
-This quickstart shows you how to stream data into and from Azure Event Hubs using the Apache Kafka protocol. You'll not change any code in the sample Kafka producer or consumer apps. You just update the configurations that the clients use to point to an Event Hubs namespace, which exposes a Kafka endpoint. You also don't build and use a Kafka cluster on your own. Instead, you'll  use the Event Hubs namespace with the Kafka endpoint.
+This quickstart shows you how to stream data into and from Azure Event Hubs using the Apache Kafka protocol. You'll not change any code in the sample Kafka producer or consumer apps. You just update the configurations that the clients use to point to an Event Hubs namespace, which exposes a Kafka endpoint. You also don't build and use a Kafka cluster on your own. Instead, you use the Event Hubs namespace with the Kafka endpoint.
 
 > [!NOTE]
 > This sample is available on [GitHub](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java)
@@ -51,13 +53,13 @@ Azure Event Hubs supports using Azure Active Directory (Azure AD) to authorize r
         1. Select the **Azure subscription** that has the VM.
         1. For **Managed identity**, select **Virtual machine**
         1. Select your virtual machine's managed identity. 
-        1. Click **Select** at the bottom of the page.
+        1. Select **Select** at the bottom of the page.
 
             :::image type="content" source="./media/event-hubs-quickstart-kafka-enabled-event-hubs/add-vm-identity.png" alt-text="Screenshot showing the Add role assignment -> Select managed identities page.":::      
     1. Select **Review + Assign**.
 
         :::image type="content" source="./media/event-hubs-quickstart-kafka-enabled-event-hubs/review-assign.png" alt-text="Screenshot showing the Add role assignment page with role assigned to VM's managed identity.":::
-1. Restart the VM and log in back to the VM for which you configured the managed identity. 
+1. Restart the VM and sign in back to the VM for which you configured the managed identity. 
 1. Clone the [Azure Event Hubs for Kafka repository](https://github.com/Azure/azure-event-hubs-for-kafka).
 1. Navigate to `azure-event-hubs-for-kafka/tutorials/oauth/java/managedidentity/consumer`.
 1. Switch to the `src/main/resources/` folder, and open `consumer.config`. Replace `namespacename` with the name of your Event Hubs namespace. 

articles/event-hubs/media/schema-registry-overview/kafka-avro.png

@@ -0,0 +1,28 @@
+---
+title: Client-side schema enforcement - Schema Registry
+description: This article provides information on using schemas in a schema registry when publishing or consuming events from Azure Event Hubs. 
+ms.topic: conceptual
+ms.date: 04/26/2023
+author: kasun04
+ms.author: kindrasiri
+---
+
+# Client-side schema enforcement 
+The information flow when you use schema registry is the same for all protocols that you use to publish or consume events from Azure Event Hubs. 
+
+The following diagram shows how the information flows when event producers and consumers use Schema Registry with the **Kafka** protocol using **Avro** serialization.  
+
+:::image type="content" source="./media/schema-registry-overview/information-flow.svg" alt-text="Image showing the Schema Registry information flow." border="false":::
+
+### Producer  
+
+1. Kafka producer application uses `KafkaAvroSerializer` to serialize event data using the specified schema. Producer application provides details of the schema registry endpoint and other optional parameters that are required for schema validation. 
+1. The serializer looks for the schema in the schema registry to serialize event data. If it finds the schema, then the corresponding schema ID is returned. You can configure the producer application to auto register the schema with the schema registry if it doesn't exist. 
+1. Then the serializer prepends the schema ID to the serialized data that is published to the Event Hubs. 
+
+### Consumer 
+
+1. Kafka consumer application uses `KafkaAvroDeserializer` to deserialize data that it receives from the event hub.
+1. The deserializer uses the schema ID (prepended by the producer) to retrieve schema from the schema registry.
+1. The deserializer uses the schema to deserialize event data that it receives from the event hub. 
+1. The schema registry client uses caching to prevent redundant schema registry lookups in the future.  

articles/event-hubs/media/schema-registry-overview/kafka-json-schema.png

@@ -0,0 +1,101 @@
+---
+title: Azure Schema Registry Concepts
+description: This article explains concepts for Azure Schema Registry in Azure Event Hubs. 
+ms.topic: conceptual
+ms.date: 04/26/2023
+author: kasun04
+ms.author: kindrasiri
+---
+
+#  Schema Registry in Azure Event Hubs
+Schema Registry in Azure Event Hubs provides you with a repository to use and manage schemas in schema-driven event streaming scenarios. 
+
+## Schema Registry components 
+
+An Event Hubs namespace can host schema groups alongside event hubs (or Kafka topics). It hosts a schema registry and can have multiple schema groups. In spite of being hosted in Azure Event Hubs, the schema registry can be used universally with all Azure messaging services and any other message or events broker. Each of these schema groups is a separately securable repository for a set of schemas. Groups can be aligned with a particular application or an organizational unit. 
+
+:::image type="content" source="./media/schema-registry-overview/elements.png" alt-text="Diagram that shows the components of Schema Registry in Azure Event Hubs." border="false":::
+
+### Schema groups
+Schema group is a logical group of similar schemas based on your business criteria. A schema group can hold multiple versions of a schema. The compatibility enforcement setting on a schema group can help ensure that newer schema versions are backwards compatible.
+
+The security boundary imposed by the grouping mechanism help ensures that trade secrets don't inadvertently leak through metadata in situations where the namespace is shared among multiple partners. It also allows for application owners to manage schemas independent of other applications that share the same namespace.
+
+### Schemas
+Schemas define the contract between producers and consumers. A schema defined in an Event Hubs schema registry helps manage the contract outside of event data, thus removing the payload overhead. A schema has a name, type (example: record, array, and so on.), compatibility mode (none, forward, backward, full), and serialization type (only Avro for now). You can create multiple versions of a schema and retrieve and use a specific version of a schema. 
+
+### Schema formats 
+Schema formats are used to determine the manner in which a schema is structured and defined, with each format outlining specific guidelines and syntax for defining the structure of the events that will be used for event streaming.
+
+#### Avro schema 
+[Avro](https://avro.apache.org/) is a popular data serialization system that uses a compact binary format and provides schema evolution capabilities. 
+
+To learn more about using Avro schema format with Event Hubs Schema Registry, see:  
+- [How to use schema registry with Kafka and Avro](schema-registry-kafka-java-send-receive-quickstart.md)
+- [ How to use Schema registry with Event Hubs .NET SDK (AMQP) and Avro.](schema-registry-dotnet-send-receive-quickstart.md)
+
+#### JSON Schema (Preview)
+[JSON Schema](https://json-schema.org/) is a standardized way of defining the structure and data types of the events. JSON Schema enables the confident and reliable use of the JSON data format in event streaming. 
+
+To learn more about using JSON schema format with Event Hubs Schema Registry, see:  
+- [How to use schema registry with Kafka and JSON Schema](schema-registry-json-schema-kafka.md)
+
+## Schema evolution 
+Schemas need to evolve with the business requirement of producers and consumers. Azure Schema Registry supports schema evolution by introducing compatibility modes at the schema group level. When you create a schema group, you can specify the compatibility mode of the schemas that you include in that schema group. When you update a schema, the change should  comply with the assigned compatibility mode and then only it creates a new version of the schema. 
+
+ > [!NOTE]
+> Schema evolution is only supported for Avro schema format only.
+
+Azure Schema Registry for Event Hubs support following compatibility modes. 
+
+### Backward compatibility
+Backward compatibility mode allows the consumer code to use a new version of schema but it can process messages with old version of the schema. When you use backward compatibility mode in a schema group, it allows following changes to be made on a schema. 
+
+- Delete fields. 
+- Add optional fields. 
+
+### Forward compatibility
+Forward compatibility allows the consumer code to use an old version of the schema but it can read messages with the new schema. Forward compatibility mode allows following changes to be made on a schema. 
+- Add fields 
+- Delete optional fields 
+
+### No compatibility
+When the ``None`` compatibility mode is used, the schema registry doesn't do any compatibility checks when you update schemas. 
+
+## Client SDKs
+
+You can use one of the following libraries to include an Avro serializer, which you can use to serialize and deserialize payloads containing Schema Registry schema identifiers and Avro-encoded data.
+
+- [.NET - Microsoft.Azure.Data.SchemaRegistry.ApacheAvro](https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/schemaregistry/Microsoft.Azure.Data.SchemaRegistry.ApacheAvro)
+- [Java - azure-data-schemaregistry-avro](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/schemaregistry/azure-data-schemaregistry-apacheavro)
+- [Python - azure-schemaregistry-avroserializer](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry-avroencoder/)
+- [JavaScript - @azure/schema-registry-avro](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/schemaregistry/schema-registry-avro)
+- [Apache Kafka](https://github.com/Azure/azure-schema-registry-for-kafka/) - Run Kafka-integrated Apache Avro serializers and deserializers backed by Azure Schema Registry. The Java client's Apache Kafka client serializer for the Azure Schema Registry can be used in any Apache Kafka scenario and with any Apache Kafka® based deployment or cloud service. 
+- **Azure CLI** - For an example of adding a schema to a schema group using CLI, see [Adding a schema to a schema group using CLI](https://github.com/Azure/azure-event-hubs/tree/master/samples/Management/CLI/AddschematoSchemaGroups).
+- **PowerShell** - For an example of adding a schema to a schema group using PowerShell, see [Adding a schema to a schema group using PowerShell](https://github.com/Azure/azure-event-hubs/tree/master/samples/Management/PowerShell/AddingSchematoSchemagroups).
+
+
+## Limits
+For limits (for example: number of schema groups in a namespace) of Event Hubs, see [Event Hubs quotas and limits](event-hubs-quotas.md)
+
+## Azure role-based access control
+When accessing the schema registry programmatically, you need to register an application in Azure Active Directory (Azure AD) and add the security principal of the application to one of the following Azure role-based access control (Azure RBAC) roles:
+
+| Role | Description | 
+| ---- | ----------- | 
+| Owner | Read, write, and delete Schema Registry groups and schemas. |
+| Contributor | Read, write, and delete Schema Registry groups and schemas. |
+| [Schema Registry Reader](../role-based-access-control/built-in-roles.md#schema-registry-reader-preview) | Read and list Schema Registry groups and schemas. |
+| [Schema Registry Contributor](../role-based-access-control/built-in-roles.md#schema-registry-reader-preview) | Read, write, and delete Schema Registry groups and schemas. |
+
+For instructions on creating registering an application using the Azure portal, see [Register an app with Azure AD](../active-directory/develop/quickstart-register-app.md). Note down the client ID (application ID), tenant ID, and the secret to use in the code. 
+
+## Next steps
+
+- To learn how to create a schema registry using the Azure portal, see [Create an Event Hubs schema registry using the Azure portal](create-schema-registry.md).
+- See the following **Schema Registry Avro client library** samples.
+    - [.NET](https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/schemaregistry/Microsoft.Azure.Data.SchemaRegistry.ApacheAvro/tests/Samples)
+    - [Java](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/schemaregistry/azure-data-schemaregistry-apacheavro/src/samples)
+    - [JavaScript](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/schemaregistry/schema-registry-avro/samples )
+    - [Python](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry-avroencoder/samples)
+    - [Kafka Avro Integration for Azure Schema Registry](https://github.com/Azure/azure-schema-registry-for-kafka/tree/master/csharp/avro/samples)