|
| 1 | +--- |
| 2 | +title: How to use certificate and Service Principal emit log to Azure Event Hubs |
| 3 | +description: Learn to setting up Azure services, particularly focusing on integrating Azure Synapse with Azure Event Hubs and Key Vault. |
| 4 | +author: jejiang |
| 5 | +ms.author: jejiang |
| 6 | +ms.reviewer: whhender |
| 7 | +ms.topic: tutorial |
| 8 | +ms.date: 03/24/2025 |
| 9 | +--- |
| 10 | + |
| 11 | +# How to use certificate and service principal emit log to Azure Event Hubs |
| 12 | + |
| 13 | +The Apache Spark diagnostic emitter extension is a library that allows Spark applications to send logs, event logs, and metrics to destinations like Azure Event Hubs, Azure Log Analytics, and Azure Storage. |
| 14 | + |
| 15 | +In this tutorial, you learn how to create required Azure resources and configure a Spark application with a certificate and service principal to emit logs, event logs, and metrics to Azure Event Hubs using the Apache Spark diagnostic emitter extension. |
| 16 | + |
| 17 | +## Prerequisites |
| 18 | + |
| 19 | +- An Azure subscription. You can also [create a free account](https://azure.microsoft.com/free/) before you get started. |
| 20 | +- [Synapse Analytics workspace](/azure/synapse-analytics/get-started-create-workspace). |
| 21 | +- [Azure Event Hubs](/azure/event-hubs/event-hubs-about). |
| 22 | +- [Azure Key Vault](/azure/key-vault/general/overview) |
| 23 | +- [App Registration](https://ms.portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade) |
| 24 | + |
| 25 | +> [!Note] |
| 26 | +> |
| 27 | +> To complete this tutorial's steps, you need to have access to a resource group for which you're assigned the Owner role. |
| 28 | +> |
| 29 | +
|
| 30 | +## Step 1. Register an application |
| 31 | + |
| 32 | +1. Sign in to the [Azure portal](https://portal.azure.com/) and go to [App registrations](/entra/identity-platform/quickstart-register-app#register-an-application). |
| 33 | +2. Create a new app registration for your Synapse workspace. |
| 34 | + |
| 35 | + :::image type="content" source="media\how-to-use-certificate-with-service-principalp-emit-log-event-hubs\create-a-new-app-registration.png" alt-text="Screenshot showing create a new app registration."::: |
| 36 | + |
| 37 | +## Step 2. Generate a certificate in Key Vault |
| 38 | + |
| 39 | +1. Navigate to Key Vault. |
| 40 | +2. Expand the **Object**, and select the **Certificates**. |
| 41 | +3. Click on **Generate/Import**. |
| 42 | + |
| 43 | + :::image type="content" source="media\how-to-use-certificate-with-service-principalp-emit-log-event-hubs\generate-a-new-certificate.png" alt-text="Screenshot showing generate a new certificate for app."::: |
| 44 | + |
| 45 | +## Step 3. Trust the certificate in the application |
| 46 | + |
| 47 | +1. Go to the app created in Step 1 -> **Manage** -> **Manifest**. |
| 48 | +2. Append the certificate details to the manifest file to establish trust. |
| 49 | + |
| 50 | + ``` |
| 51 | + "trustedCertificateSubjects": [ |
| 52 | + { |
| 53 | + "authorityId": "00000000-0000-0000-0000-000000000001", |
| 54 | + "subjectName": "Your-Subject-of-Certificate", |
| 55 | + "revokedCertificateIdentifiers": [] |
| 56 | + } |
| 57 | + ] |
| 58 | + ``` |
| 59 | +
|
| 60 | + :::image type="content" source="media\how-to-use-certificate-with-service-principalp-emit-log-event-hubs\trust-the-certificate.png" alt-text="Screenshot showing trust the certificate in the application."::: |
| 61 | +
|
| 62 | +## Step 4. Assign Azure Event Hubs Data Sender Role |
| 63 | +
|
| 64 | +1. In Azure Event Hubs, navigate to Access control (IAM). |
| 65 | +2. Assign the Azure Event Hubs data sender role to the application (service principal). |
| 66 | +
|
| 67 | + :::image type="content" source="media\how-to-use-certificate-with-service-principalp-emit-log-event-hubs\assign-azure-event-hubs-data-sender-role.png" alt-text="Screenshot showing assign Azure event hubs data sender role."::: |
| 68 | +
|
| 69 | +## Step 5. Create a linked service in Synapse |
| 70 | +
|
| 71 | +1. In Synapse Analytics workspace, go to **Manage** -> **linked service**. |
| 72 | +2. Create a new **linked Service** in Synapse to connect to **Key Vault**. |
| 73 | +
|
| 74 | + :::image type="content" source="media\how-to-use-certificate-with-service-principalp-emit-log-event-hubs\create-a-linked-service-in-synapse.png" alt-text="Screenshot showing create a linked service in synapse."::: |
| 75 | +
|
| 76 | +## Step 6. Assign reader role to linked service in Key Vault |
| 77 | +
|
| 78 | +1. Get the workspace managed identity ID from the linked service. The **managed identity name** and **object ID** for the linked service is under **Edit linked service**. |
| 79 | +
|
| 80 | + :::image type="content" source="media\how-to-use-certificate-with-service-principalp-emit-log-event-hubs\managed-identity-name-and-object-id.png" alt-text="Screenshot showing managed identity name and object ID are in edit linked service."::: |
| 81 | +
|
| 82 | +2. In **Key Vault**, assign the linked service a **Reader** role. |
| 83 | +
|
| 84 | +## Step 7. Configure with a linked service |
| 85 | +
|
| 86 | +Gather the following values and add to the Apache Spark configuration. |
| 87 | +
|
| 88 | +- **<EMITTER_NAME>**: The name for the emmiter. |
| 89 | +- **<CERTIFICATE_NAME>**: The certificate name that you generated in the key vault. |
| 90 | +- **<LINKED_SERVICE_NAME>**: The Azure Key vault linked service name. |
| 91 | +- **<EVENT_HUB_HOST_NAME>**: The Azure Event Hubs host name, you can find it in Azure Event Hubs Namespace -> Overview -> Host name. |
| 92 | +- **<SERVICE_PRINCIPAL_TENANT_ID>**: The service principal tenant ID, you can find it in App registrations -> your app name -> Overview -> Directory (tenant) ID |
| 93 | +- **<SERVICE_PRINCIPAL_CLIENT_ID>**: The service principal client ID, you can find it in registrations -> your app name -> Overview -> Application(client) ID |
| 94 | +- **<EVENT_HUB_ENTITY_PATH>**: The Azure Event Hubs entity path, you can find it in Azure Event Hubs Namespace -> Overview -> Host name. |
| 95 | +
|
| 96 | +``` |
| 97 | + "spark.synapse.diagnostic.emitters": <EMITTER_NAME>, |
| 98 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.type": "AzureEventHub", |
| 99 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.categories": "DriverLog,ExecutorLog,EventLog,Metrics", |
| 100 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.certificate.keyVault.certificateName": <CERTIFICATE_NAME>", |
| 101 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.certificate.keyVault.linkedService": <LINKED_SERVICE_NAME>, |
| 102 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.hostName": <EVENT_HUB_HOST_NAME>, |
| 103 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.tenantId": <SERVICE_PRINCIPAL_TENANT_ID>, |
| 104 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.clientId": <SERVICE_PRINCIPAL_CLIENT_ID>, |
| 105 | + "spark.synapse.diagnostic.emitter.<EMITTER_NAME>.entityPath": <EVENT_HUB_ENTITY_PATH> |
| 106 | +``` |
| 107 | +
|
| 108 | +## Step 8. Submit an Apache Spark application and view the logs and metrics |
| 109 | +
|
| 110 | +You can use the Apache Log4j library to write custom logs. |
| 111 | +
|
| 112 | +Example for Scala: |
| 113 | +
|
| 114 | +```scala |
| 115 | +%%spark |
| 116 | +val logger = org.apache.log4j.LogManager.getLogger("com.contoso.LoggerExample") |
| 117 | +logger.info("info message") |
| 118 | +logger.warn("warn message") |
| 119 | +logger.error("error message") |
| 120 | +//log exception |
| 121 | +try { |
| 122 | + 1/0 |
| 123 | + } catch { |
| 124 | + case e:Exception =>logger.warn("Exception", e) |
| 125 | +} |
| 126 | +// run job for task level metrics |
| 127 | +val data = sc.parallelize(Seq(1,2,3,4)).toDF().count() |
| 128 | +``` |
| 129 | + |
| 130 | +Example for PySpark: |
| 131 | + |
| 132 | +```python |
| 133 | +%%pyspark |
| 134 | +logger = sc._jvm.org.apache.log4j.LogManager.getLogger("com.contoso.PythonLoggerExample") |
| 135 | +logger.info("info message") |
| 136 | +logger.warn("warn message") |
| 137 | +logger.error("error message") |
| 138 | +``` |
| 139 | + |
0 commit comments