Merge pull request #233085 from bwren/waf

Azure Monitor WAF service guide

Author: v-ccolin

articles/azure-monitor/best-practices-cost.md

@@ -0,0 +1,50 @@
+---
+title: Best practices for Azure Monitor Logs
+description: Provides a template for a Well-Architected Framework (WAF) article specific to Log Analytics workspaces in Azure Monitor.
+ms.topic: conceptual
+author: bwren
+ms.author: bwren
+ms.date: 03/29/2023
+ms.reviewer: bwren
+---
+
+# Best practices for Azure Monitor Logs
+This article provides architectural best practices for Azure Monitor Logs. The guidance is based on the five pillars of architecture excellence described in [Azure Well-Architected Framework](/azure/architecture/framework/).
+
+
+
+## Reliability
+In the cloud, we acknowledge that failures happen. Instead of trying to prevent failures altogether, the goal is to minimize the effects of a single failing component. Use the following information to minimize failure of your Log Analytics workspaces and to protect the data they collect.
+
+[!INCLUDE [waf-logs-reliability](includes/waf-logs-reliability.md)]
+
+
+## Security
+Security is one of the most important aspects of any architecture. Azure Monitor provides features to employ both the principle of least privilege and defense-in-depth. Use the following information to maximize the security of your Log Analytics workspaces and ensure that only authorized users access collected data.
+
+[!INCLUDE [waf-logs-security](includes/waf-logs-security.md)]
+
+
+## Cost optimization
+Cost optimization refers to ways to reduce unnecessary expenses and improve operational efficiencies. You can significantly reduce your cost for Azure Monitor by understanding your different configuration options and opportunities to reduce the amount of data that it collects. See [Azure Monitor cost and usage](usage-estimated-costs.md) to understand the different ways that Azure Monitor charges and how to view your monthly bill.
+
+> [!NOTE]
+> See [Optimize costs in Azure Monitor](best-practices-cost.md) for cost optimization recommendations across all features of Azure Monitor.
+
+[!INCLUDE [waf-logs-cost](includes/waf-logs-cost.md)]
+
+
+## Operational excellence
+Operational excellence refers to operations processes required keep a service running reliably in production. Use the following information to minimize the operational requirements for supporting Log Analytics workspaces.
+
+[!INCLUDE [waf-logs-operation](includes/waf-logs-operation.md)]
+
+
+## Performance efficiency
+Performance efficiency is the ability of your workload to scale to meet the demands placed on it by users in an efficient manner. Use the following information to ensure that your Log Analytics workspaces and log queries are configured for maximum performance.
+
+[!INCLUDE [waf-logs-performance](includes/waf-logs-performance.md)]
+
+## Next step
+
+- [Get best practices for a complete deployment of Azure Monitor](best-practices.md).

articles/azure-monitor/best-practices-logs.md

@@ -506,7 +506,7 @@ sections:
         answer: |
           Yes, for experimental use. In the basic pricing plan, your application can send a certain allowance of data each month free of charge. The free allowance is large enough to cover development and publishing an app for a few users. You can set a cap to prevent more than a specified amount of data from being processed.
           
-          Larger volumes of telemetry are charged by the gigabyte. We provide some tips on how to [limit your charges](best-practices-cost.md#data-collection).
+          Larger volumes of telemetry are charged by the gigabyte. We provide some tips on how to [limit your charges](best-practices-cost.md#application-insights).
           
           The Enterprise plan incurs a charge for each day that each web server node sends telemetry. It's suitable if you want to use Continuous Export on a large scale.
           

articles/azure-monitor/faq.yml

@@ -0,0 +1,29 @@
+---
+author: bwren
+ms.author: bwren
+ms.service: azure-monitor
+ms.topic: include
+ms.date: 03/30/2023
+---
+
+### Design checklist
+
+> [!div class="checklist"]
+> - Configure pricing tier for the amount of data that each Log Analytics workspace typically collects.
+> - Configure tables used for debugging, troubleshooting, and auditing as Basic Logs.
+> - Configure data retention and archiving.
+> - Regularly analyze collected data to identify trends and anomalies.
+> - Create an alert when data collection is high.
+> - Consider a daily cap as a preventative measure to ensure that you don't exceed a particular budget.
+
+### Configuration recommendations
+
+| Recommendation | Benefit |
+|:---|:---|
+| Determine whether to combine your operational data and your security data in the same Log Analytics workspace. | Since all data in a Log Analytics workspace is subject to Microsoft Sentinel pricing if Sentinel is enabled, there may be cost implications to combining this data. See [Design a Log Analytics workspace architecture](../logs/workspace-design.md) for details on making this decision for your environment balancing it with criteria in other pillars. |
+| Configure pricing tier for the amount of data that each Log Analytics workspace typically collects. | By default, Log Analytics workspaces will use pay-as-you-go pricing with no minimum data volume. If you collect enough data, you can significantly decrease your cost by using a [commitment tier](../logs/cost-logs.md#commitment-tiers), which allows you to commit to a daily minimum of data collected in exchange for a lower rate. If you collect enough data across workspaces in a single region, you can link them to a [dedicated cluster](../logs/logs-dedicated-clusters.md) and combine their collected volume using [cluster pricing](../logs/cost-logs.md#dedicated-clusters).<br><br>See [Azure Monitor Logs cost calculations and options](../logs/cost-logs.md) for details on commitment tiers and guidance on determining which is most appropriate for your level of usage. See [Usage and estimated costs](../usage-estimated-costs.md#usage-and-estimated-costs) to view estimated costs for your usage at different pricing tiers.  |
+| Configure data retention and archiving. | There is a charge for retaining data in a Log Analytics workspace beyond the default of 31 days (90 days if Sentinel is enabled on the workspace and 90 days for Application insights data). Consider your particular requirements for having data readily available for log queries. You can significantly reduce your cost by configuring [Archived Logs](../logs/data-retention-archive.md), which allows you to retain data for up to seven years and still access it occasionally using [search jobs](../logs/search-jobs.md) or [restoring a set of data](../logs/restore.md) to the workspace. |
+| Configure tables used for debugging, troubleshooting, and auditing as Basic Logs. | Tables in a Log Analytics workspace configured for [Basic Logs](../logs/basic-logs-configure.md) have a lower ingestion cost in exchange for limited features and a charge for log queries. If you query these tables infrequently and don't use them for alerting, this query cost can be more than offset by the reduced ingestion cost. |
+| Regularly analyze collected data to identify trends and anomalies.  | Use [Log Analytics workspace insights](../logs/log-analytics-workspace-insights-overview.md) to periodically review the amount of data collected in your workspace. In addition to helping you understand the amount of data collected by different sources, it will identify anomalies and upward trends in data collection that could result in excess cost. Further analyze data collection using methods in [Analyze usage in Log Analytics workspace](../logs/analyze-usage.md) to determine if there's additional configuration that can decrease your usage further. This is particularly important when you add a new set of data sources, such as a new set of virtual machines or onboard a new service. |
+| Create an alert when data collection is high. | To avoid unexpected bills, you should be [proactively notified anytime you experience excessive usage](../logs/analyze-usage.md#send-alert-when-data-collection-is-high). Notification allows you to address any potential anomalies before the end of your billing period. |
+| Consider a daily cap as a preventative measure to ensure that you don't exceed a particular budget. | A [daily cap](../logs/daily-cap.md) disables data collection in a Log Analytics workspace for the rest of the day after your configured limit is reached. This shouldn't be used as a method to reduce costs as described in [When to use a daily cap](../logs/daily-cap.md#when-to-use-a-daily-cap).<br><br>If you do set a daily cap, in addition to [creating an alert when the cap is reached](../logs/log-analytics-workspace-health.md#view-log-analytics-workspace-health-and-set-up-health-status-alerts),ensure that you also [create an alert rule to be notified when some percentage has been reached (90% for example)](../logs/analyze-usage.md#send-alert-when-data-collection-is-high). This gives you an opportunity to investigate and address the cause of the increased data before the cap shuts off data collection. |

articles/azure-monitor/includes/waf-logs-cost.md

@@ -0,0 +1,23 @@
+---
+author: bwren
+ms.author: bwren
+ms.service: azure-monitor
+ms.topic: include
+ms.date: 03/30/2023
+---
+
+### Design checklist
+
+> [!div class="checklist"]
+> - Design a workspace architecture with the minimal number of workspaces to meet your business requirements.
+> - Use Log Analytics workspace insights to track the health and performance of your Log Analytics workspaces.
+> - Create alert rules to be proactively notified of operational issues in the workspace.
+
+### Configuration recommendations
+
+| Recommendation | Benefit |
+|:---|:---|
+| Design a workspace architecture with the minimal number of workspaces to meet your business requirements. | A single or at least minimal number of workspaces will maximize your operational efficiency since all of your operational and security data will be located in a single location increasing your visibility into potential issues and making patterns easier to identify. You minimize your requirement for [cross-workspace queries](../logs/cross-workspace-query.md) and need to manage the configuration and data for fewer workspaces.<br><br>You may have requirements for multiple workspaces, such as multiple tenants or regulatory compliance requiring multiple regions, but you should balance those requirements against a goal of creating the minimal number of workspaces. See [Design a Log Analytics workspace architecture](../logs/workspace-design.md) for a list of decision criteria. |
+| Use Log Analytics workspace insights to track the health and performance of your Log Analytics workspaces.  | [Log Analytics workspace insights](../logs/workspace-design.md) provides a unified view of the usage, performance, health, agents, queries, and change log for all your workspaces. Review this information on a regular basis to track the health and operation of each of your workspaces. |
+| Create alert rules to be proactively notified of operational issues in the workspace. | Each workspace has an [operation table](../logs/monitor-workspace.md) that logs important activities affecting workspace. Create alert rules based on this table to be proactively notified when an operational issue occurs. You can use [recommended alerts for the workspace](../logs/log-analytics-workspace-health.md) to simplify the creation of the most critical alert rules. |
+

articles/azure-monitor/includes/waf-logs-operation.md

@@ -0,0 +1,18 @@
+---
+author: bwren
+ms.author: bwren
+ms.service: azure-monitor
+ms.topic: include
+ms.date: 03/30/2023
+---
+
+### Design checklist
+
+> [!div class="checklist"]
+> - Configure log query auditing and use Log Analytics workspace insights to identify slow and inefficient queries.
+
+### Configuration recommendations
+
+| Recommendation | Benefit |
+|:---|:---|
+| Configure log query auditing and use Log Analytics workspace insights to identify slow and inefficient queries. | [Log query auditing](../logs/query-audit.md) stores the compute time required to run each query and the time until results are returned. [Log Analytics workspace insights](../logs/log-analytics-workspace-insights-overview.md#query-audit-tab) uses this data to list potentially inefficient queries in your workspace. Consider rewriting these queries to improve their performance. Refer to [Optimize log queries in Azure Monitor](../logs/query-optimization.md) for guidance on optimizing your log queries. |

articles/azure-monitor/includes/waf-logs-performance.md

@@ -0,0 +1,30 @@
+---
+author: bwren
+ms.author: bwren
+ms.service: azure-monitor
+ms.topic: include
+ms.date: 03/30/2023
+---
+
+Log Analytics workspaces offer a high degree of reliability without any design decisions. Conditions where a temporary loss of access to the workspace can result in data loss are often mitigated by features of other Azure Monitor components such as data buffering with the Azure Monitor agent.
+
+The reliability situations to consider for Log Analytics workspaces are availability of the workspace and protection of collected data in the rare case of failure of an Azure datacenter or region. There is currently no standard feature for failover between workspaces in different regions, but there are strategies that you can use if you have particular requirements for availability or compliance.
+
+Some availability features require a dedicated cluster. Since this requires a commitment of at least 500 GB per day from all workspaces in the same region, reliability will not typically be your primary criteria for including dedicated clusters in your design.
+
+### Design checklist
+
+> [!div class="checklist"]
+> - If you collect enough data for a dedicated cluster, create a dedicated cluster in an availability zone.
+> - If you require the workspace to be available in the case of a region failure, or you don't collect enough data for a dedicated cluster, configure data collection to send critical data to multiple workspaces in different regions.
+> - If you require data to be protected in the case of datacenter or region failure, configure data export from the workspace to save data in an alternate location.
+> - Create a health status alert rule for your Log Analytics workspace.
+ 
+### Configuration recommendations
+
+| Recommendation | Benefit |
+|:---|:---|
+| If you collect enough data, create a dedicated cluster in a region that supports availability zones. | Workspaces linked to a [dedicated cluster](../logs/logs-dedicated-clusters.md) located in a region that supports [availability zones](../logs/availability-zones.md#data-resilience---supported-regions) remain available if a datacenter fails. |
+| If you require the workspace to be available in the case of a region failure, or you don't collect enough data for a dedicated cluster, configure data collection to send critical data to multiple workspaces in different regions. | Configure your data sources to send to multiple workspaces in different regions. For example, configure DCRs for multiple workspaces for Azure Monitor agent running on virtual machines, and multiple diagnostic settings to collection resource logs from Azure resources. This configuration results in duplicate ingestion and retention charges so only use it for critical data.<br><br>Even though the data will be available in the alternate workspace in case of failure, resources that rely on the data such as alerts and workbooks wouldn't know to use this workspace. Consider storing ARM templates for critical resources with configuration for the alternate workspace in Azure DevOps or as disabled [policies](../../governance/policy/overview.md) that can quickly be enabled in a failover scenario. |
+| If you require data to be protected in the case of datacenter or region failure, configure data export from the workspace to save data in an alternate location. | The [data export feature of Azure Monitor](../logs/logs-data-export.md) allows you to continuously export data sent to specific tables to Azure storage where it can be retained for extended periods. Use [Azure Storage redundancy options](../../storage/common/storage-redundancy.md#redundancy-in-a-secondary-region) including GRS and GZRS to replicate this data to other regions. If you require export of [tables that aren't supported by data export](../logs/logs-data-export.md?tabs=portal#limitations) then you can use other methods of exporting data including Logic apps to protect your data. This is primarily a solution to meet compliance for data retention since the data can be difficult to analyze and restore back to the workspace. |
+| Create a health status alert rule for your Log Analytics workspace. | A [health status alert](../logs/log-analytics-workspace-health.md#view-log-analytics-workspace-health-and-set-up-health-status-alerts) will proactively notify you if a workspace becomes unavailable because of a datacenter or regional failure. |