Added a new Topic

Added a new Topic

Author: sreekzz

articles/hdinsight/selective-logging-analysis-azure-logs.md

@@ -0,0 +1,170 @@
+---
+title: Use selective logging with a script action in Azure HDInsight clusters
+description: Learn how to use the selective logging feature with a script action to monitor logs.
+ms.service: azure-hdinsight
+ms.topic: how-to
+ms.custom: references-regions
+ms.date: 09/13/2023
+---
+
+# Use selective logging with a script action in Azure HDInsight
+
+[Azure Monitor Logs](../azure-monitor/logs/log-query-overview.md) is an Azure Monitor service that monitors your cloud and on-premises environments. The monitoring helps maintain their availability and performance. 
+
+Azure Monitor Logs collects data generated by resources in your cloud, resources in on-premises environments, and other monitoring tools. It uses the data to provide analysis across multiple sources. To get the analysis, you enable the selective logging feature by using a script action for HDInsight in the Azure portal.
+
+## About selective logging
+
+Selective logging is a part of the overall monitoring system in Azure. After you connect your cluster to a Log Analytics workspace and enable selective logging, you can see logs and metrics like HDInsight security logs, Yarn Resource Manager, and system metrics. You can monitor workloads and see how they're affecting cluster stability. 
+
+Selective logging allows you to enable or disable all the tables, or enable selected tables, in the Log Analytics workspace. You can adjust the source type for each table.
+
+> [!NOTE]  
+> If Log Analytics is reinstalled in a cluster, you'll have to disable all the tables and log types again. Reinstallation resets all the configuration files to their original state.
+
+## Considerations for script actions
+
+* The monitoring system uses the Metadata Server Daemon (a monitoring agent) and Fluentd for collecting logs by using a unified logging layer.
+* Selective logging uses a script action to disable or enable tables and their log types. Because selective logging doesn't open any new ports or change any existing security settings, there are no security changes.
+* The script action runs in parallel on all specified nodes and changes the configuration files for disabling or enabling tables and their log types.
+
+## Prerequisites
+
+* A Log Analytics workspace. You can think of this workspace as a unique Azure Monitor Logs environment with its own data repository, data sources, and solutions. For instructions, see  [Create a Log Analytics workspace](../azure-monitor/vm/monitor-virtual-machine.md).
+* An Azure HDInsight cluster. Currently, you can use the selective logging feature with the following HDInsight cluster types:
+   * Hadoop
+   * HBase
+   * Interactive Query
+   * Spark
+
+For instructions on how to create an HDInsight cluster, see [Get started with Azure HDInsight](hadoop/apache-hadoop-linux-tutorial-get-started.md).  
+
+## Enable or disable logs by using a script action for multiple tables and log types
+
+1. Go to **Script actions** in your cluster and select **Submit new** to start the process of creating a script action. 
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/select-submit-script-action.png" alt-text="Screenshot that shows the button for starting the process of creating a script action.":::
+
+    The **Submit script action** pane appears.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/submit-script-action-window.png" alt-text="Screenshot that shows the pane for submitting a script action.":::
+
+1. For the script type, select **Custom**.
+1. Name the script. For example: **Disable two tables and two sources**.
+1. The Bash script URI must be a link to [selectiveLoggingScript.sh](https://hdiconfigactions.blob.core.windows.net/log-analytics-patch/selectiveLoggingScripts/selectiveLoggingScript.sh).
+1. Select all the node types that apply for the cluster. The options are head node, worker node, and ZooKeeper node.
+1. Define the parameters. For example:
+   - Spark: `spark HDInsightSparkLogs:SparkExecutorLog --disable`
+   - Interactive Query: `interactivehive HDInsightSparkLogs:SparkExecutorLog --enable`
+   - Hadoop: `hadoop HDInsightSparkLogs:SparkExecutorLog --disable`
+   - HBase: `hbase HDInsightSparkLogs: HDInsightHBaseLogs   --enable`
+
+   For more information, see the [Parameter syntax](#parameter-syntax) section.
+
+1. Select **Create**.
+1. After a few minutes, a green check mark appears next to your script action history. It means the script has successfully run.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/enable-table-and-log-types.png" alt-text="Screenshot that shows a successful run of a script to enable tables and log types.":::
+
+You'll see your changes in the Log Analytics workspace.
+
+## Troubleshooting
+
+### No changes appear in the Log Analytics workspace
+
+If you submit your script action but there are no changes in the Log Analytics workspace:
+
+1. Under **Dashboards**, select **Ambari home** to check the debug information.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/select-dashboard-ambari-home.png" alt-text="Screenshot that shows the location of the Ambari home dashboard.":::
+
+1. Select the **Settings** button.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/ambari-dash-board.png" alt-text="Screenshot that shows the Settings button.":::
+
+1. Select your latest script run at the top of the list of background operations.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/background-operations.png" alt-text="Screenshot that shows background operations.":::
+
+1. Verify the script run status in all the nodes individually.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/background-operations-all.png" alt-text="Screenshot that shows the script run status for hosts.":::
+
+1. Check that the parameter syntax from the parameter syntax section is correct.
+1. Check that the Log Analytics workspace is connected to the cluster and that Log Analytics monitoring is turned on.
+1. Check that you selected the **Persist this script action to rerun when new nodes are added to the cluster** checkbox for the script action that you ran.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/script-action-persists.png" alt-text="Screenshot that shows the checkbox for persisting a script action.":::
+
+1. See if a new node has been added to the cluster recently.
+
+    > [!NOTE]  
+    > For the script to run in the latest cluster, the script must persist.
+   
+1. Make sure that you selected all the node types that you wanted for the script action.
+
+    :::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/select-node-types.png" alt-text="Screenshot that shows selected node types.":::
+
+### The script action failed
+
+If the script action shows a failure status in the script action history:
+
+1. Check that the parameter syntax from the parameter syntax section is correct.
+1. Check that the script link is correct. It should be: `https://hdiconfigactions.blob.core.windows.net/log-analytics-patch/selectiveLoggingScripts/selectiveLoggingScript.sh`.
+
+## Table names
+
+For a complete listing of table names for different log types (sources), see [Azure Monitor Logs tables](monitor-hdinsight-reference.md#azure-monitor-logs-tables).
+
+## Parameter syntax
+
+Parameters define the cluster type, table names, source names, and action.
+
+:::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/parameter-syntax-box.png" alt-text="Screenshot that shows the parameter syntax box.":::
+
+A parameter contains three parts:
+- Cluster type
+- Tables and log types
+- Action (either `--disable` or `--enable`)
+
+### Syntax for multiple tables
+
+When you have multiple tables, they're separated with a comma. For example:
+
+`spark HDInsightSecurityLogs, HDInsightAmbariSystemMetrics --disable`
+
+`hbase HDInsightSecurityLogs, HDInsightAmbariSystemMetrics --enable`
+
+### Syntax for multiple source types or log types
+
+When you have multiple source types or log types, they're separated with a space.
+
+To disable a source, write the table name that contains the log types, followed by a colon and then the real log type name:
+
+`TableName :  LogTypeName`
+
+For example, assume that `spark HDInsightSecurityLogs` is a table that has two log types: `AmbariAuditLog` and `AuthLog`. To disable both the log types, the correct syntax would be:
+
+`spark HDInsightSecurityLogs: AmbariAuditLog AuthLog --disable`
+
+### Syntax for multiple tables and source types
+
+If you need to disable two tables and two source types, use the following syntax:
+
+- Spark: `InteractiveHiveMetastoreLog` log type in the `HDInsightHiveAndLLAPLogs` table
+- Hbase: `InteractiveHiveHSILog` log type in the `HDInsightHiveAndLLAPLogs` table
+- Hadoop: `HDInsightHiveAndLLAPMetrics` table
+- Hadoop: `HDInsightHiveTezAppStats` table
+
+Separate the tables with a comma. Denote sources by using a colon after the table name in which they reside. 
+
+The correct parameter syntax for these cases would be:
+
+```
+interactivehive HDInsightHiveAndLLAPLogs: InteractiveHiveMetastoreLog, HDInsightHiveAndLLAPMetrics, HDInsightHiveTezAppStats, HDInsightHiveAndLLAPLogs: InteractiveHiveHSILog --enable 
+```
+
+## Next steps
+
+* [Query Azure Monitor Logs to monitor HDInsight clusters](hdinsight-hadoop-oms-log-analytics-use-queries.md)
+* [Monitor cluster availability with Apache Ambari and Azure Monitor Logs](./hdinsight-cluster-availability.md)