diff --git a/README.md b/README.md index 2e01640..70578d9 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ Last updated: 2025-05-02 - [Networking](#networking) - [Monitoring and Observability](#monitoring-and-observability) - [Cost Management](#cost-management) +- [Best Practices](#best-practices) @@ -196,6 +197,18 @@ Click to read more about [Microsoft Purview for Fabric - Overview](./Purview-Fab > Click to read [Billing Report - Overview](./Cost-Management/BillingReport.md), and [Budget & Alerts - Overview](./Cost-Management/BudgetAlerts.md). +## Best Practices + +- [Azure Data Factory (ADF) - Best Practices Overview](./Workloads-Specific/DataFactory/BestPractices.md) +- [Data Engineering - Best Practices Overview]() - in progress +- [Data Warehouse - Best Practices Overview]() - in progress +- [Data Science - Best Practices Overview]() - in progress +- [Real-Time Intelligence - Best Practices Overview]() - in progress +- [Power Bi - Best Practices Overview](./Workloads-Specific/PowerBi/BestPractices.md) +- [Copilot - Best Practices Overview]() - in progress +- [Purview - Best Practices Overview]() - in progress +- [OneLake - Best Practices Overview]() - in progress +

Total Visitors

Visitor Count diff --git a/Workloads-Specific/DataFactory/BestPractices.md b/Workloads-Specific/DataFactory/BestPractices.md index 13ec549..5337aad 100644 --- a/Workloads-Specific/DataFactory/BestPractices.md +++ b/Workloads-Specific/DataFactory/BestPractices.md @@ -6,7 +6,7 @@ Costa Rica [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/) [brown9804](https://github.com/brown9804) -Last updated: 2025-04-21 +Last updated: 2025-05-02 ---------- @@ -26,46 +26,38 @@ Last updated: 2025-04-21
Table of Content (Click to expand) -- [Architecture examples](#architecture-examples) -- [Best Practices for ADF Pipelines](#best-practices-for-adf-pipelines) - - [Clear Pipeline Structure](#clear-pipeline-structure) - - [Example Pipeline Structure](#example-pipeline-structure) - - [Parameterization](#parameterization) - - [Incremental Loading](#incremental-loading) - - [Use Timestamps](#use-timestamps) - - [Change Data Capture CDC](#change-data-capture-cdc) - - [Delta Loads](#delta-loads) - - [Partitioning](#partitioning) - - [Error Handling and Monitoring](#error-handling-and-monitoring) - - [a. Use If Condition Activity](#a-use-if-condition-activity) - - [b. Configure Activity Fault Tolerance](#b-configure-activity-fault-tolerance) - - [c. Custom Error Handling: Use Web Activity for error handling](#c-custom-error-handling-use-web-activity-for-error-handling) - - [d. Pipeline Monitoring: Monitor activity runs.](#d-pipeline-monitoring-monitor-activity-runs) - - [Security Measures](#security-measures) - - [Use Azure Key Vault](#use-azure-key-vault) - - [Store Secrets](#store-secrets) - - [Access Policies](#access-policies) - - [Secure Access](#secure-access) - - [Rotate Secrets](#rotate-secrets) - - [Source Control](#source-control) - - [Resource Management](#resource-management) - - [Testing and Validation](#testing-and-validation) - - [Documentation](#documentation) - - [Regular Updates](#regular-updates) - - [Performance Tuning](#performance-tuning) +- [Clear Pipeline Structure](#clear-pipeline-structure) + - [Example Pipeline Structure](#example-pipeline-structure) +- [Parameterization](#parameterization) +- [Incremental Loading](#incremental-loading) + - [Use Timestamps](#use-timestamps) + - [Change Data Capture CDC](#change-data-capture-cdc) + - [Delta Loads](#delta-loads) + - [Partitioning](#partitioning) +- [Error Handling and Monitoring](#error-handling-and-monitoring) + - [a. Use If Condition Activity](#a-use-if-condition-activity) + - [b. Configure Activity Fault Tolerance](#b-configure-activity-fault-tolerance) + - [c. Custom Error Handling: Use Web Activity for error handling](#c-custom-error-handling-use-web-activity-for-error-handling) + - [d. Pipeline Monitoring: Monitor activity runs.](#d-pipeline-monitoring-monitor-activity-runs) +- [Security Measures](#security-measures) +- [Use Azure Key Vault](#use-azure-key-vault) + - [Store Secrets](#store-secrets) + - [Access Policies](#access-policies) + - [Secure Access](#secure-access) + - [Rotate Secrets](#rotate-secrets) +- [Source Control](#source-control) +- [Resource Management](#resource-management) +- [Testing and Validation](#testing-and-validation) +- [Documentation](#documentation) +- [Regular Updates](#regular-updates) +- [Performance Tuning](#performance-tuning) - [Recommended Training Modules on Microsoft Learn](#recommended-training-modules-on-microsoft-learn) +- [Architecture examples](#architecture-examples)
-## Architecture examples - -image - -image - -## Best Practices for ADF Pipelines -### Clear Pipeline Structure +## Clear Pipeline Structure > Ensure your pipelines are well-organized and easy to understand. @@ -78,7 +70,7 @@ Last updated: 2025-04-21 | **Organized Layout** | Arrange activities in a logical sequence and avoid overlapping lines. | - Place activities in a left-to-right or top-to-bottom flow to visually represent the data flow.
- Group related activities together and use containers for better organization. | | **Error Handling and Logging**| Include error handling and logging activities to capture and manage errors. | - Add a Web Activity to log errors to a monitoring system.
- Use Try-Catch blocks to handle errors gracefully and ensure the pipeline continues running. | -#### Example Pipeline Structure +### Example Pipeline Structure > Pipeline: CopySalesDataPipeline @@ -135,8 +127,8 @@ graph TD image -### Parameterization -> +## Parameterization + > Use parameters to make your pipelines more flexible and easier to manage. | **Best Practice** | **Description** | **Example** | @@ -146,8 +138,8 @@ graph TD | **Global Parameters** | Use global parameters for values that are used across multiple pipelines. | - Define a global parameter for a storage account name used in various pipelines.
- Create a global parameter for a common API key used across multiple pipelines.
- Use a global parameter for a base URL that is referenced in multiple activities. | | **Parameterize Datasets** | Parameterize datasets to handle different data sources or destinations. | - Create a dataset with a parameterized file path to handle different file names dynamically.
- Use parameters in datasets to switch between different databases or tables.
- Define parameters for connection strings to dynamically connect to different data sources. | -### Incremental Loading -> +## Incremental Loading + > Implement incremental data loading to improve efficiency. | **Best Practice** | **Description** | **Example** | @@ -157,7 +149,7 @@ graph TD | **Delta Loads** | Perform delta loads to update only the changed data instead of full loads. | - Use a query to fetch only the rows that have changed since the last load.
- Implement a mechanism to track changes, such as a version number or a change flag. | | **Partitioning** | Partition large datasets to improve performance and manageability. | - Partition data by date or another logical key to facilitate incremental loading.
- Use partitioned tables in your data warehouse to improve query performance and manageability. | -#### Use Timestamps +### Use Timestamps > Implement incremental loading using timestamps to load only new or changed data. @@ -175,8 +167,8 @@ graph TD - After loading the data, update the watermark table with the latest timestamp. - Use a Stored Procedure activity to update the `LastLoadedTimestamp` in the watermark table. -#### Change Data Capture (CDC) -> +### Change Data Capture (CDC) + > Utilize CDC to capture and load only the changes made to the source data. 1. **Enable CDC on Source Table**: @@ -189,8 +181,8 @@ graph TD - Use a ForEach activity to process each change. - Inside the ForEach activity, use Copy Data activities to apply the changes to the destination. -#### Delta Loads -> +### Delta Loads + > Perform delta loads to update only the changed data instead of full loads. 1. **Track Changes**: @@ -203,8 +195,8 @@ graph TD - Use a Copy Data activity to load only the changed data. - After loading, reset the `ChangeFlag` to 0. -#### Partitioning -> +### Partitioning + > Partition large datasets to improve performance and manageability. 1. **Partition Your Data**: @@ -217,8 +209,8 @@ graph TD - Use a ForEach activity to process each partition. - Inside the ForEach activity, use a Copy Data activity to load data for each partition. -### Error Handling and Monitoring -> +## Error Handling and Monitoring + > Set up robust error handling and monitoring to quickly identify and resolve issues. | **Best Practice** | **Description** | **Example** | @@ -228,7 +220,7 @@ graph TD | **Alerts and Notifications** | Set up alerts and notifications to monitor pipeline runs and failures. | - Use Azure Monitor to create alerts for failed pipeline runs and send email notifications.
- Configure alerts to trigger SMS notifications for critical pipeline failures.
- Set up a Logic App to send Slack notifications when a pipeline fails. | | **Custom Logging** | Implement custom logging to capture detailed error information. | - Use a Web Activity to log errors to an external logging service or database.
- Implement an Azure Function to log detailed error information and call it from the pipeline.
- Use a Set Variable activity to capture error details and write them to a log file in Azure Blob Storage. | -#### a. **Use If Condition Activity** +### a. **Use If Condition Activity** 1. **Create a Pipeline**: - Open Microsoft Fabric and navigate to Azure Data Factory. @@ -255,7 +247,7 @@ graph TD image -#### b. **Configure Activity Fault Tolerance** +### b. **Configure Activity Fault Tolerance** 1. **Set Retry Policy**: - Select an activity within your pipeline. @@ -267,7 +259,7 @@ graph TD image -#### c. **Custom Error Handling**: Use Web Activity for error handling +### c. **Custom Error Handling**: Use Web Activity for error handling - Add a Web Activity to your pipeline. @@ -277,7 +269,7 @@ graph TD image -#### d. **Pipeline Monitoring**: Monitor activity runs +### d. **Pipeline Monitoring**: Monitor activity runs - In the ADF monitoring interface, navigate to the `Monitor` section, if you don't see it click on `...`. - Check the status of individual activities within your pipelines for success, failure, and skipped activities. Or search for any specific pipeline. @@ -287,8 +279,8 @@ graph TD image -### Security Measures -> +## Security Measures + > Apply security best practices to protect your data. | **Best Practice** | **Description** | **Example** | @@ -298,8 +290,8 @@ graph TD | **Network Security** | Use virtual networks and private endpoints to secure data access. | - Configure ADF to use a private endpoint for accessing data in a storage account.
- Set up a virtual network (VNet) to isolate and secure ADF resources.
- Use Network Security Groups (NSGs) to control inbound and outbound traffic to ADF. | | **Audit Logs** | Enable auditing to track access and changes to ADF resources. | - Use Azure Monitor to collect and analyze audit logs for ADF activities.
- Enable diagnostic settings to send logs to Azure Log Analytics, Event Hubs, or a storage account.
- Regularly review audit logs to detect and respond to unauthorized access or changes. | -### Use Azure Key Vault -> +## Use Azure Key Vault + > Store sensitive information such as connection strings, passwords, and API keys in Azure Key Vault to enhance security and manage secrets efficiently. | **Best Practice** | **Description** | **Example** | @@ -309,7 +301,7 @@ graph TD | **Secure Access** | Use managed identities to securely access Key Vault secrets. | - Configure ADF to use its managed identity to retrieve secrets from Key Vault.
- Enable managed identity for ADF and grant it access to Key Vault secrets.
- Use managed identities to avoid storing credentials in code or configuration files. | | **Rotate Secrets** | Regularly rotate secrets to enhance security. | - Update secrets in Key Vault periodically and update references in ADF.
- Implement a process to rotate secrets automatically using Azure Automation or Logic Apps.
- Notify relevant teams when secrets are rotated to ensure they update their configurations. | -#### Store Secrets +### Store Secrets > Store sensitive information such as connection strings, passwords, and API keys in Key Vault. @@ -333,8 +325,8 @@ graph TD image -#### Access Policies -> +### Access Policies + > Configure access policies to control who can access secrets. 1. **Set Up Access Policies in Key Vault**: @@ -346,7 +338,7 @@ graph TD - Define access policies to allow only specific users or applications to retrieve secrets. - Example: Grant access to specific roles such as `DataFactoryContributor` for managing secrets. -#### Secure Access +### Secure Access > Use managed identities to securely access Key Vault secrets. @@ -355,8 +347,8 @@ graph TD - In the Key Vault, add an access policy to grant the Data Factory managed identity access to the required secrets. - Example: Grant `Get` and `List` permissions to the managed identity. -#### Rotate Secrets -> +### Rotate Secrets + > Regularly rotate secrets to enhance security. 1. **Update Secrets in Key Vault**: @@ -369,10 +361,9 @@ graph TD - Ensure that relevant teams are notified when secrets are rotated. - Example: Use Logic Apps to send email notifications when secrets are updated. -### Source Control +## Source Control > Benefits of Git Integration:
-> > - **Version Control**: Track and audit changes, and revert to previous versions if needed.
> - **Collaboration**: Multiple team members can work on the same project simultaneously.
> - **Incremental Saves**: Save partial changes without publishing them live.
@@ -406,8 +397,8 @@ graph TD - Use pull requests to review and merge changes from feature branches to the collaboration branch. - Collaborate with team members through code reviews and comments. -### Resource Management -> +## Resource Management + > Optimize resource usage to improve performance and reduce costs. | **Best Practice** | **Description** | **Example** | @@ -417,8 +408,8 @@ graph TD | **Cost Management** | Implement cost management practices to control expenses. | - Use Azure Cost Management to monitor and manage ADF costs.
- Set budgets and alerts to avoid unexpected expenses.
- Review and optimize the use of Data Integration Units (DIUs) to balance cost and performance. | | **Resource Tagging** | Tag resources for better organization and cost tracking. | - Apply tags to ADF resources to categorize and track costs by project or department.
- Use tags to identify and manage resources associated with specific business units.
- Implement tagging policies to ensure consistent resource tagging across the organization. | -### Testing and Validation -> +## Testing and Validation + > Regularly test and validate your pipelines to ensure they work as expected. | **Best Practice** | **Description** | **Example** | @@ -428,8 +419,8 @@ graph TD | **Validation Activities** | Use validation activities to check data quality and integrity. | - Add a validation activity to verify the row count or data format after a Copy Data activity.
- Implement data quality checks to ensure data accuracy and completeness.
- Use custom scripts or functions to validate complex data transformations. | | **Automated Testing** | Automate testing processes to ensure consistency and reliability. | - Use Azure DevOps pipelines to automate the testing of ADF pipelines.
- Schedule automated tests to run after each deployment or code change.
- Integrate automated testing with CI/CD pipelines to ensure continuous validation. | -### Documentation -> +## Documentation + > Maintain comprehensive documentation for your pipelines. | **Best Practice** | **Description** | **Example** | @@ -439,8 +430,8 @@ graph TD | **Annotations** | Use annotations within ADF to provide context and explanations. | - Add annotations to activities to describe their function and any important details.
- Use comments to explain complex logic or business rules within the pipeline.
- Highlight key parameters and settings with annotations for easy reference. | | **Knowledge Sharing** | Share documentation with the team to ensure everyone is informed. | - Use a shared platform like SharePoint or Confluence to store and share documentation.
- Conduct regular training sessions to keep the team updated on best practices.
- Encourage team members to contribute to and update the documentation. | -### Regular Updates -> +## Regular Updates + > Keep your pipelines and ADF environment up to date. | **Best Practice** | **Description** | **Example** | @@ -450,8 +441,8 @@ graph TD | **Dependency Management** | Keep dependencies up to date to avoid compatibility issues. | - Update linked services and datasets to use the latest versions of data sources.
- Regularly review and update external dependencies like libraries and APIs.
- Ensure compatibility between ADF and other integrated services. | | **Security Patches** | Apply security patches promptly to protect against vulnerabilities. | - Monitor security advisories and apply patches to ADF and related services.
- Implement a patch management process to ensure timely updates.
- Conduct regular security assessments to identify and address vulnerabilities. | -### Performance Tuning -> +## Performance Tuning + > Continuously monitor and tune performance. | **Best Practice** | **Description** | **Example** | @@ -471,6 +462,14 @@ graph TD - [A categorized list of Azure Data Factory tutorials by scenarios](https://learn.microsoft.com/en-us/azure/data-factory/data-factory-tutorials) - [Full list of Data Factory trainings](https://learn.microsoft.com/en-us/training/browse/?expanded=azure&products=azure-data-factory) +## Architecture examples + +> Consider lakehouse or warehouse for storage: + +image + +image +

Total Visitors

Visitor Count diff --git a/Workloads-Specific/PowerBi/BestPractices.md b/Workloads-Specific/PowerBi/BestPractices.md new file mode 100644 index 0000000..d0e2c23 --- /dev/null +++ b/Workloads-Specific/PowerBi/BestPractices.md @@ -0,0 +1,168 @@ +# Power Bi - Best Practices Overview + +Costa Rica + +[![GitHub](https://badgen.net/badge/icon/github?icon=github&label)](https://github.com) +[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/) +[brown9804](https://github.com/brown9804) + +Last updated: 2025-05-02 + +---------- + +
+List of References (Click to expand) + +- [Tips for designing a great Power BI dashboard – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/create-reports/service-dashboards-design-tips) +- [Power BI guidance documentation – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/guidance/) +- [Row-level security (RLS) with Power BI – Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/security/service-admin-row-level-security) +- [Row-level security (RLS) guidance in Power BI Desktop – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/guidance/rls-guidance) +- [Azure DevOps build pipeline integration with Power BI Desktop projects – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/developer/projects/projects-build-pipelines) +- [Power BI Desktop projects Azure DevOps integration – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/developer/projects/projects-azdo) +- [Optimize ribbon in Power BI Desktop – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/create-reports/desktop-optimize-ribbon) +- [DirectQuery optimization scenarios with the Optimize ribbon in Power BI Desktop – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/create-reports/desktop-optimize-ribbon-scenarios) +- [What is Microsoft Fabric for Power BI service users?](https://learn.microsoft.com/en-us/power-bi/fundamentals/fabric-power-bi) +- [Tutorial: Microsoft Fabric for Power BI users – Microsoft Learn](https://learn.microsoft.com/en-us/power-bi/fundamentals/fabric-get-started) +- [Automate deployment pipelines with APIs for Power BI items – Microsoft Learn](https://learn.microsoft.com/en-us/fabric/cicd/deployment-pipelines/pipeline-automation) +- [Understand star schema and the importance for Power BI](https://learn.microsoft.com/en-us/power-bi/guidance/star-schema) + +
+ +
+Table of Content (Click to expand) + +- [Clear Dashboard and Report Structure](#clear-dashboard-and-report-structure) + - [Example Report Layout](#example-report-layout) +- [Parameterization and Dynamic Content](#parameterization-and-dynamic-content) +- [Incremental Data Refresh](#incremental-data-refresh) + - [Optimized Refresh Schedules](#optimized-refresh-schedules) +- [Data Modeling and DAX Optimization](#data-modeling-and-dax-optimization) +- [Security and Governance](#security-and-governance) + - [Row-Level Security RLS](#row-level-security-rls) +- [Source Control and Collaboration](#source-control-and-collaboration) +- [Performance Tuning](#performance-tuning) +- [Documentation and Maintenance](#documentation-and-maintenance) +- [Deploying Reports with DevOps Best Practices](#deploying-reports-with-devops-best-practices) + +
+ +## Clear Dashboard and Report Structure + +> Ensure that your Power BI dashboards and reports are organized, visually cohesive, and provide clear insights at a glance. + +| **Best Practice** | **Description** | **Example** | +|--------------------------------------|------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Consistent Naming Conventions** | Use clear and descriptive names for reports, pages, visuals, measures, and datasets. | Instead of generic names like `Report1`, prefer descriptive names such as `SalesPerformanceReport`. | +| **Modular Report Design** | Organize your reports by separating key performance indicators (KPIs), detailed analysis, and context. | Design separate pages for overviews, analytics details, and contextual information as your audience drills down. | +| **Visual Consistency** | Apply a uniform theme and layout using Power BI themes to maintain brand consistency and readability. | Follow [Tips for designing a great Power BI dashboard](https://learn.microsoft.com/en-us/power-bi/create-reports/service-dashboards-design-tips) for layout and visual emphasis best practices. | +| **Annotations and Tooltips** | Document complex visual logic and calculations via in-report annotations and dynamic tooltips. | Include tooltips that explain data sources or DAX logic to enhance report understandability. | + +### Example Report Layout + +> **Report**: SalesPerformanceReport + +1. **Data Model**: Use a star schema to separate fact tables from dimension tables for clarity and efficiency. +2. **DAX Measures**: Implement clear DAX measures with inline comments to describe the underlying logic. +3. **Visualizations**: Combine various visuals (charts, tables, slicers) to provide interactive data insights. +4. **Dashboard Overview**: Summarize key metrics and trends on a dedicated overview page. + +
+ Centered Image +
+ +From [Understand star schema and the importance for Power BI](https://learn.microsoft.com/en-us/power-bi/guidance/star-schema) + +## Parameterization and Dynamic Content + +> Enhance report flexibility by using parameters and dynamic expressions. + +| **Best Practice** | **Description** | **Example** | +|----------------------------------------|------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Power Query Parameters** | Use parameters to dynamically adjust data sources and filter conditions during data ingestion. | Define date range parameters in Power Query to allow reports to automatically update based on user configurations. | +| **Dynamic DAX Expressions** | Create DAX measures that change in response to slicer selections or other user inputs. | Implement dynamic calculations using variables in DAX so that measures (like Year-over-Year growth) adjust automatically per user selection. | +| **Dynamic Drill-throughs** | Build drill-through URLs and links dynamically for a seamless navigation experience. | Concatenate strings and field values in DAX to direct users to detailed reports or external dashboards. | + +## Incremental Data Refresh + +> Optimize your data refresh process by updating only the data that has changed. + +| **Best Practice** | **Description** | **Example** | +|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Enable Incremental Refresh** | Set up incremental refresh policies to refresh only new or modified data, reducing load time and resource usage. | Use the [Data refresh in Power BI](https://learn.microsoft.com/en-us/power-bi/connect-data/refresh-data) guidelines to configure incremental refresh for large datasets. | +| **Data Partitioning** | Partition your data by logical segments (e.g., date ranges) to target refresh operations more efficiently. | Partition sales or transaction data by month or quarter so that only the most recent partitions are refreshed during scheduled updates. | + +> Click to read [Incremental Refresh for Reporting - Overview](./Workloads-Specific/PowerBi/IncrementalRefresh.md) + +### Optimized Refresh Schedules + +- **Staggered Refreshes**: Schedule refresh operations during off-peak hours to balance resource consumption. +- **Monitoring Refresh Performance**: Follow best practices in [Configure scheduled refresh](https://learn.microsoft.com/en-us/power-bi/connect-data/refresh-scheduled-refresh) to monitor and troubleshoot scheduled refresh cycles. + + https://github.com/user-attachments/assets/61780cc9-5778-4c55-8f05-ffbaedd7802c + +## Data Modeling and DAX Optimization + +> Build robust data models and optimize DAX calculations to ensure high performance and maintainability. + +| **Best Practice** | **Description** | **Example** | +|--------------------------------------|---------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Star Schema Implementation** | Structure your data using a star schema to improve clarity and query performance. | Separate fact tables (e.g., Sales) from dimension tables (e.g., Date, Customer) as recommended by [Power BI guidance documentation](https://learn.microsoft.com/en-us/power-bi/guidance/). | +| **Efficient DAX Coding** | Utilize variables and avoid deeply nested functions to streamline DAX calculations. | Replace nested CALCULATE functions with variables:
VAR TotalSales = SUM('Sales'[Amount]) RETURN TotalSales to simplify logic and improve performance. | +| **Reduce Cardinality When Possible** | Pre-aggregate data where applicable to reduce the detail in fact tables and improve efficiency. | Aggregate detailed transaction logs to daily totals if high granularity is not necessary, as covered in the [Optimization guide for Power BI](https://learn.microsoft.com/en-us/power-bi/guidance/power-bi-optimization). | + +## Security and Governance + +> Safeguard data by implementing robust security measures and managing data access appropriately. + +### Row-Level Security (RLS) + +| **Best Practice** | **Description** | **Example** | +|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Define RLS in Power BI Desktop** | Use Power BI Desktop to create roles and restrict data at the row level. | Implement RLS by defining roles such as `SalesManager` and applying DAX filters (e.g., `[Region] = USERPRINCIPALNAME()`) as described in [Row-level security (RLS) guidance in Power BI Desktop](https://learn.microsoft.com/en-us/power-bi/guidance/rls-guidance). | +| **Test RLS Thoroughly** | Validate security settings using the “View as Role” feature before publishing your reports. | Verify RLS filters by simulating different user roles to ensure users can only view allowed data. | +| **Optimize RLS Implementation** | Apply RLS filters preferably on dimension tables so that filtering propagates efficiently through relationships. | Structure RLS so that filters on the Customer or Region tables automatically restrict the related fact table data. | + +## Source Control and Collaboration + +> Leverage version control systems and collaborative workspaces for efficient report deployment and teamwork. + +| **Best Practice** | **Description** | **Example** | +|---------------------------------------|----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Git-based Version Control** | Use Git repositories (via Azure DevOps or GitHub) to manage and version your Power BI Desktop projects. | Follow the [Azure DevOps build pipeline integration with Power BI Desktop projects](https://learn.microsoft.com/en-us/power-bi/developer/projects/projects-build-pipelines) guidance for setting up continuous integration. | +| **Collaborative Workspaces** | Configure shared workspaces within the Power BI service for team collaboration and controlled deployment. | Integrate your workspace with Git as described in [Power BI Desktop projects Azure DevOps integration](https://learn.microsoft.com/en-us/power-bi/developer/projects/projects-azdo) to manage team contributions. | +| **Change Documentation** | Maintain a detailed changelog or README file to document major updates and decision rationales. | Use Git commit messages and an in-repository README to capture changes and ensure transparency across development cycles. + +## Performance Tuning + +> Enhance report responsiveness and reduce resource load by optimizing visuals and query performance. + +| **Best Practice** | **Description** | **Example** | +|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Optimize Visuals and Interactions** | Limit the number of visuals per report page and simplify complex interactions to improve performance. | Use [Optimize ribbon in Power BI Desktop](https://learn.microsoft.com/en-us/power-bi/create-reports/desktop-optimize-ribbon) to pause visuals and control query execution during design. | +| **DirectQuery Optimization** | Use features from the Optimize ribbon to streamline DirectQuery interactions and reduce redundant queries. | Follow methods in [DirectQuery optimization scenarios with the Optimize ribbon in Power BI Desktop](https://learn.microsoft.com/en-us/power-bi/create-reports/desktop-optimize-ribbon-scenarios) to adjust visual refresh behavior. | + +## Documentation and Maintenance + +> Keep your reports self-explanatory and maintain a centralized repository for ongoing support and future enhancements. + +| **Best Practice** | **Description** | **Example** | +|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **In-Report Annotations** | Use text boxes, tooltips, and in-report comments to document data sources, transformations, and calculation logic. | Include explanatory notes within report pages as encouraged in the [Power BI guidance documentation](https://learn.microsoft.com/en-us/power-bi/guidance/). | +| **Scheduled Reviews and Updates** | Periodically review report performance, refresh schedules, and security settings to keep the solution optimized and current. | Adopt a routine review cycle based on feedback and usage metrics provided by Power BI service administration tools. | +| **Centralized Repository** | Store and version-control all project artifacts (PBIX files, change logs, documentation) in a secure repository. | Use Azure DevOps or another Git-based repository, following the practices laid out in the source control sections above. + +## Deploying Reports with DevOps Best Practices + +> Automate and streamline deployment processes using CI/CD pipelines integrated within your Power BI environment. + +| **Best Practice** | **Description** | **Example** | +|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Automated Build Pipelines** | Integrate Git repositories with Azure DevOps to automatically validate and deploy Power BI content. | Refer to [Azure DevOps build pipeline integration with Power BI Desktop projects](https://learn.microsoft.com/en-us/power-bi/developer/projects/projects-build-pipelines) for guidance. | +| **Deployment Pipelines via APIs** | Use the Power BI deployment pipelines APIs to automate report deployment steps and ensure quality across environments. | Implement automated deployments as described in [Automate deployment pipelines with APIs for Power BI items](https://learn.microsoft.com/en-us/fabric/cicd/deployment-pipelines/pipeline-automation). | +| **Continuous Integration** | Incorporate quality gates and automated testing within your CI/CD process to catch issues early in the development cycle. | Utilize Git-based integration and automated tests as shown in [Power BI Desktop projects Azure DevOps integration](https://learn.microsoft.com/en-us/power-bi/developer/projects/projects-azdo) for best practices. | + + +
+

Total Visitors

+ Visitor Count +
diff --git a/Workloads-Specific/PowerBi/IncrementalRefresh.md b/Workloads-Specific/PowerBi/IncrementalRefresh.md index a6ce326..3cdd161 100644 --- a/Workloads-Specific/PowerBi/IncrementalRefresh.md +++ b/Workloads-Specific/PowerBi/IncrementalRefresh.md @@ -1,11 +1,11 @@ -# Power Bi: Incremental Refresh for Reporting - Overview +# Incremental Refresh for Reporting - Overview Costa Rica [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/) [brown9804](https://github.com/brown9804) -Last updated: 2025-04-21 +Last updated: 2025-05-02 ----------