Skip to content

Commit 47c6815

Browse files
authored
adding medallion arch
1 parent 5dd5ff5 commit 47c6815

File tree

1 file changed

+21
-3
lines changed

1 file changed

+21
-3
lines changed

Workloads-Specific/DataWarehouse/BestPractices.md

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,19 @@ Last updated: 2025-05-03
1515
<details>
1616
<summary><b>List of References</b> (Click to expand)</summary>
1717

18+
- [Ingest data into the Warehouse](https://learn.microsoft.com/en-us/fabric/data-warehouse/ingest-data)
19+
- [Performance guidelines in Fabric Data Warehouse](https://learn.microsoft.com/en-us/fabric/data-warehouse/guidelines-warehouse-performance)
20+
1821
</details>
1922

2023
<details>
2124
<summary><b>Table of Content</b> (Click to expand)</summary>
2225

26+
- [Sample Warehouse Environment](#sample-warehouse-environment)
27+
- [Structured Warehouse Implementation](#structured-warehouse-implementation)
28+
- [Interactive Notebooks for Data Warehousing](#interactive-notebooks-for-data-warehousing)
29+
- [Using Mirroring to Your Benefit](#using-mirroring-to-your-benefit)
30+
2331
</details>
2432

2533
<div align="center">
@@ -41,12 +49,11 @@ Create a warehouse solution that segments data as follows:
4149
- Silver Layer: Applies data cleansing, validation, and enrichment.
4250
- Gold Layer: Produces analytics-ready data using optimized storage formats like Parquet or Delta Lake, with partitioning by date or region. Integrate metadata catalogs and RBAC controls for added governance.
4351

44-
> Here is a [reference of a medallion architecture using only Fabric](./Workloads-Specific/DataWarehouse/Medallion_Archuitecture/). <br/>
45-
> If you need to handle `complex data transformations and large-scale data processing`, you can use our combined solution of **Fabric + Databricks**. This powerful combination leverages the strengths of both platforms to provide a robust data processing pipeline. This workshop on [Fabric with Databricks for Data Analytics](https://microsoft.github.io/TechExcel-Fabric-with-Databricks-for-Data-Analytics/) offers a comprehensive step-by-step guide on developing Medallion Architecture using Fabric and Databricks. <br/>
52+
> Here is a [reference of a medallion architecture using only Fabric](./Workloads-Specific/DataWarehouse/Medallion_Archuitecture/). If you need to handle `complex data transformations and large-scale data processing`, you can use our combined solution of **Fabric + Databricks**. This powerful combination leverages the strengths of both platforms to provide a robust data processing pipeline. This workshop on [Fabric with Databricks for Data Analytics](https://microsoft.github.io/TechExcel-Fabric-with-Databricks-for-Data-Analytics/) offers a comprehensive step-by-step guide on developing Medallion Architecture using Fabric and Databricks. <br/>
4653
4754
| Medallion Architecture using only Fabric | Medallion Architecture Fabric + Databricks |
4855
| --- | --- |
49-
| <img width="550" alt="image" src="https://github.com/user-attachments/assets/b4394d54-9bb0-453b-abf8-cfaaa8e532d2" /> | <img width="550" alt="image" src="https://github.com/user-attachments/assets/c866098c-ffd1-4438-bc77-565786c91601">
56+
| <img width="550" alt="image" src="https://github.com/user-attachments/assets/b4394d54-9bb0-453b-abf8-cfaaa8e532d2" /> | <img width="550" alt="image" src="https://github.com/user-attachments/assets/c866098c-ffd1-4438-bc77-565786c91601"> |
5057

5158
## Interactive Notebooks for Data Warehousing
5259

@@ -61,7 +68,18 @@ Create notebooks that are segmented into distinct sections:
6168

6269
> Mirroring offers a modern, efficient way to continuously and seamlessly access and ingest data from operational databases or data warehouses. It works by replicating a snapshot of the source database into OneLake, and then keeping that replica in near real-time sync with the original. This ensures that your data is always up to date and readily available for analytics or downstream processing. `As part of the value offering, each Fabric compute SKU includes a built-in allowance of free Mirroring storage, proportional to the compute capacity you provision. For example, provisioning an F64 SKU grants you 64 terabytes of free Mirroring storage. You only begin incurring OneLake storage charges if your mirrored data exceeds this free limit or if the compute capacity is paused.` Click [here](https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/?msockid=38ec3806873362243e122ce086486339) to read more about it.
6370
71+
<div align="center">
72+
<img src="https://github.com/user-attachments/assets/ed868665-1823-42ff-9cd7-d0ee3310c184" alt="Centered Image" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
73+
</div>
6474

75+
| **Mirroring Option** | Details |
76+
|--------------------------------------------------|--------------------|
77+
| **Mirrored Azure SQL Database** | Configure a mirrored Azure SQL Database with geo-redundancy and automatic failover. For example, use Azure’s built-in replication to maintain a secondary copy that seamlessly takes over during primary instance outages, ensuring continuous data availability. |
78+
| **Mirrored Snowflake** | Deploy a Snowflake mirror by setting up data replication between your primary instance and a secondary environment. Regularly validate synchronization and monitor rollback capabilities to confirm that the mirror remains current and can support operations during failover or testing cycles. |
79+
| **Mirrored Azure Cosmos DB** | Configure an Azure Cosmos DB mirroring setup in preview mode that replicates data across multiple regions. Test the environment by simulating high-load queries and failover events to ensure that global access is maintained with minimal latency. |
80+
| **Mirrored Azure Database for PostgreSQL** | Set up a mirrored Azure Database for PostgreSQL in its preview configuration. Create read replicas with continuous synchronization, perform failover drills, and track replication latency to guarantee that the mirrored instance maintains data integrity and high availability during operational stress. |
81+
| **Mirrored Azure SQL Managed Instance** | Configure an Azure SQL Managed Instance in a mirrored setup using strategies like log shipping or transactional replication. Monitor key performance metrics to ensure that replication latency is minimal, and the mirror is capable of supporting a swift transition during outages or maintenance windows. |
82+
| **Mirrored Database** | Set up a mirrored database configuration that synchronizes periodically with a primary instance. Schedule automated tests and synchronization checks, and simulate failover events to validate that the data remains consistent, with built-in alerts and monitoring demonstrating the mirror’s readiness for production use. |
6583

6684
<div align="center">
6785
<h3 style="color: #4CAF50;">Total Visitors</h3>

0 commit comments

Comments
 (0)