You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Workloads-Specific/DataWarehouse/Medallion_Architecture/README.md
+19-15Lines changed: 19 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,13 +23,12 @@ Last updated: 2025-05-03
23
23
24
24
-[Overview](#overview)
25
25
-[Demo](#demo)
26
-
-[Step 1: Set Up Your Environment](#step-1-set-up-your-environment)
27
-
-[Step 2: Ingest Data into the Bronze Layer](#step-2-ingest-data-into-the-bronze-layer)
28
-
-[Step 3: Transform Data in the Silver Layer](#step-3-transform-data-in-the-silver-layer)
29
-
-[Step 4: Curate Data in the Gold Layer](#step-4-curate-data-in-the-gold-layer)
30
-
-[Step 5: Set Up Pipelines for Orchestration](#step-5-set-up-pipelines-for-orchestration)
31
-
-[Step 6: Enable Data Access for Reporting](#step-6-enable-data-access-for-reporting)
32
-
26
+
-[Step 1: Set Up Your Environment](#step-1-set-up-your-environment)
27
+
-[Step 2: Ingest Data into the Bronze Layer](#step-2-ingest-data-into-the-bronze-layer)
28
+
-[Step 3: Transform Data in the Silver Layer](#step-3-transform-data-in-the-silver-layer)
29
+
-[Step 4: Curate Data in the Gold Layer](#step-4-curate-data-in-the-gold-layer)
30
+
-[Step 5: Set Up Pipelines for Orchestration](#step-5-set-up-pipelines-for-orchestration)
31
+
-[Step 6: Enable Data Access for Reporting](#step-6-enable-data-access-for-reporting)
33
32
34
33
</details>
35
34
@@ -49,14 +48,14 @@ Last updated: 2025-05-03
49
48
> [!IMPORTANT]
50
49
> If you are not able to see the `auto-create report` option neither `copilot` be aware you need to enable AI features in your tenant, click [here](https://github.com/brown9804/MicrosoftCloudEssentialsHub/blob/main/0_Azure/2_AzureAnalytics/0_Fabric/demos/6_PBiCopilot.md#tenant-configuration) to see how.
Implementing a medallion architecture provides several benefits:
58
+
60
59
-**Data Quality**: By organizing data into layers, you can apply quality checks and transformations in a structured manner, ensuring that the data in the Gold layer is reliable and ready for analysis.
61
60
-**Scalability**: The architecture allows you to scale your data processing pipelines independently for each layer, providing flexibility and efficiency.
62
61
-**Performance**: The Gold layer is optimized for performance, which means that your reporting and analytics queries will run faster.
@@ -79,14 +78,13 @@ Implementing a medallion architecture provides several benefits:
79
78
- Click on `Workspaces`, then select either your existing workspace or create a new one by clicking `New Workspace`:
80
79
- Provide a name and other required details, then create the workspace.
> Besides using Data pipelines to bring your SQL information, you can also leverage Microsoft Fabric's mirrored SQL capability. This feature allows you to create a mirrored copy of your SQL database, improving data availability, reliability, and disaster recovery. By maintaining a synchronized copy of your database in a different location, it ensures that your data is always accessible, even in the event of a failure or outage.
> `For example, both Azure SQL Database and Microsoft Fabric are Microsoft products. However, the concept of outbound connections still applies because the data is moving from one service (Azure SQL Database) to another service (Microsoft Fabric), even though they are both within the Microsoft ecosystem. This movement of data is considered outbound because it is leaving the Azure SQL Database environment and entering the Microsoft Fabric environment`. <br/> <br/>
156
154
> Under the Zero Trust Architecture, both inbound and outbound connections are treated with the same level of scrutiny and security protocols. This means that whether the connection is inbound or outbound, it is subject to strict verification processes to ensure it is safe and authorized. Key principles of Zero Trust include verification of every access request, least privilege access, continuous monitoring, and micro-segmentation. By applying these principles, Azure ensures that both inbound and outbound connections are secure, reducing the risk of unauthorized access and data breaches.
1. **Create Pipelines**: Create pipelines to automate the movement of data from the Bronze layer to the Silver layer, andfrom the Silver layer to the Gold layer.
252
255
- In Data Factory, create a new pipeline.
253
256
- Add a copy activity to move data from the `raw_Bronze` lakehouse to the `cleansed_Silver` lakehouse.
- Consider the frequency of data updates and the latency that is acceptable for your use case.
258
261
259
262
### Step 6: Enable Data Access for Reporting
263
+
260
264
1. **Configure SQL Analytics Endpoint**:
261
265
- Validate if you have the SQL Analytics Endpoint configured, you can review it from workspace view, primarly is required for Gold layer to be accessible to your reporting tools.
0 commit comments