You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Use the selective logging feature with a script action in Azure HDInsight
10
+
# Use selective logging with a script action in Azure HDInsight
11
11
12
12
[Azure Monitor Logs](../azure-monitor/logs/log-query-overview.md) is an Azure Monitor service that monitors your cloud and on-premises environments. The monitoring helps maintain their availability and performance.
13
13
14
-
Azure Monitor Logs collects data generated by resources in your cloud, resources in on-premises environments, and other monitoring tools. It uses the data to provide analysis across multiple sources. To get the analysis, you enable the selective logging feature by using a script action in the Azure portal in HDInsight.
14
+
Azure Monitor Logs collects data generated by resources in your cloud, resources in on-premises environments, and other monitoring tools. It uses the data to provide analysis across multiple sources. To get the analysis, you enable the selective logging feature by using a script action for HDInsight in the Azure portal.
15
15
16
16
## About selective logging
17
17
18
-
Selective logging is a part of Azure overall monitoring system. After you connect your cluster to a Log Analytics workspace and enable selective logging, you can see logs and metrics like HDInsight security logs, Yarn Resource Manager, and system metrics. You can monitor workloads and see how they're affecting cluster stability.
18
+
Selective logging is a part of the overall monitoring system in Azure. After you connect your cluster to a Log Analytics workspace and enable selective logging, you can see logs and metrics like HDInsight security logs, Yarn Resource Manager, and system metrics. You can monitor workloads and see how they're affecting cluster stability.
19
19
20
-
Selective logging allows you to enable or disable all the tables of selected tables in the Log Analytics workspace. You can adjust the source type for each table.
20
+
Selective logging allows you to enable or disable all the tables, or enable selected tables, in the Log Analytics workspace. You can adjust the source type for each table.
21
21
22
22
> [!NOTE]
23
23
> If Log Analytics is reinstalled in a cluster, you'll have to disable all the tables and log types again. Reinstallation resets all the configuration files to their original state.
@@ -55,7 +55,7 @@ For instructions on how to create an HDInsight cluster, see [Get started with Az
55
55
1. Select all the node types that apply for the cluster. The options are head node, worker node, and Zookeeper node.
@@ -64,7 +64,7 @@ For instructions on how to create an HDInsight cluster, see [Get started with Az
64
64
1. Select **Create**.
65
65
1. After a few minutes, a green check mark appears next to your script action history. It means the script has successfully run.
66
66
67
-
:::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/enable-table-and-log-types.png" alt-text="Screenshot showing enable table and log types.":::
67
+
:::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/enable-table-and-log-types.png" alt-text="Screenshot that shows a successful run of a script to enable tables and log types.":::
68
68
69
69
You'll see the desired changes in the Log Analytics workspace.
70
70
@@ -101,7 +101,7 @@ If you submit your script action but there are no changes in the Log Analytics w
101
101
> [!NOTE]
102
102
> For the script to run in the latest cluster, the script must persist.
103
103
104
-
1. Make sure that you seleted all the node types that you wanted for the script action.
104
+
1. Make sure that you selected all the node types that you wanted for the script action.
105
105
106
106
:::image type="content" source="./media/hdinsight-hadoop-oms-selective-log-analytics-tutorial/select-node-types.png" alt-text="Screenshot that shows selected node types.":::
107
107
@@ -116,7 +116,7 @@ If the script action shows a **Failed** status in the script action history:
116
116
117
117
### Spark cluster
118
118
119
-
Different log types(sources) inside **Spark** tables
119
+
The following table names are for different log types(sources) inside Spark tables.
120
120
121
121
| Source number | Table name | Log types | Description |
122
122
| --- | --- | --- | --- |
@@ -128,38 +128,38 @@ Different log types(sources) inside **Spark** tables
128
128
| 6. | HDInsightHadoopAnd YarnMetrics | No log types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
129
129
| 7. | HDInsightOozieLogs | Oozie | This table contains all logs generated from the Oozie framework. |
130
130
131
-
### Interactive query cluster
131
+
### Interactive Query cluster
132
132
133
-
Different log types(sources) inside **interactive query** tables
133
+
The following table names are for different log types(sources) inside Interactive Query tables.
134
134
135
135
| Source number | Table name | Log types | Description |
136
136
| --- | --- | --- | --- |
137
137
| 1. | HDInsightAmbariClusterAlerts | No log types | This table contains Ambari cluster alerts from each node in the cluster (except for edge nodes). Each alert is a record in this table. |
138
138
| 2. | HDInsightAmbariSystem Metrics | No log types | This table contains system metrics collected from Ambari. The metrics now come from each node in the cluster (except for edge nodes) instead of just the two head nodes. Each metric is now a column, and each metric is reported once per record. |
139
-
| 3. | HDInsightHadoopAndYarnLogs |**Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node:** NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
139
+
| 3. | HDInsightHadoopAndYarnLogs |**Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node**: NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
140
140
| 4. | HDInsightHadoopAndYarnMetrics | No log types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
141
141
| 5. | HDInsightHiveAndLLAPLogs |**Head node**: InteractiveHiveHSILog, InteractiveHiveMetastoreLog, ZeppelinLog | This table contains logs generated from Hive, LLAP, and their related components: WebHCat and Zeppelin. |
142
142
| 6. | HDInsightHiveAndLLAPmetrics | No log types | This table contains JMX metrics from the Hive and LLAP frameworks. It contains all the same JMX metrics as the old Custom Logs tables. It contains one metric per record. |
143
143
| 7. | HDInsightHiveTezAppStats | No log types |
144
-
| 8. | HDInsightSecurityLogs |**Head node**: AmbariAuditLog, AuthLog **Zookeeper node, worker node**: AuthLog | This table contains records from the Ambari audit and authentication Logs. |
144
+
| 8. | HDInsightSecurityLogs |**Head node**: AmbariAuditLog, AuthLog **Zookeeper node, worker node**: AuthLog | This table contains records from the Ambari audit and authentication logs. |
145
145
146
146
### HBase cluster
147
147
148
-
Different log types(sources) inside **HBase** tables
148
+
The following table names are for different log types(sources) inside HBase tables.
149
149
150
150
| Source number | Table name | Log types | Description |
151
151
| --- | --- | --- | --- |
152
152
| 1. | HDInsightAmbariClusterAlerts | No other log types | This table contains Ambari cluster alerts from each node in the cluster (except for edge nodes). Each alert is a record in this table.
153
153
| 2. | HDInsightAmbariSystem Metrics | No other log types | This table contains system metrics collected from Ambari. The metrics now come from each node in the cluster (except for edge nodes) instead of just the two head nodes. Each metric is now a column, and each metric is reported once per record. |
154
-
| 3. | HDInsightHadoopAndYarnLogs |**Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node:** NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
155
-
| 4. | HDInsightSecurityLogs |**Head node:** AmbariAuditLog, AuthLog **Worker node:** AuthLog **ZooKeper node:** AuthLog | This table contains records from the Ambari audit and authentication logs. |
156
-
| 5. | HDInsightHBaseLogs |**Head node**: HDFSGarbageCollectorLog, HDFSNameNodeLog **Worker node:** PhoenixServerLog, HBaseRegionServerLog, HBaseRestServerLog **Zookeeper node:** HBaseMasterLog | This table contains logs from HBase and its related components: Phoenix and HDFS. |
157
-
| 6. | HDInsightHBaseMetrics | No log types | This table contains JMX metrics from HBase. It contains all the same JMX metrics from the tables listed in the Old Schema column. In contrast from the old tables, each row contains one metric. |
154
+
| 3. | HDInsightHadoopAndYarnLogs |**Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node**: NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
155
+
| 4. | HDInsightSecurityLogs |**Head node**: AmbariAuditLog, AuthLog **Worker node**: AuthLog **Zookeeper node**: AuthLog | This table contains records from the Ambari audit and authentication logs. |
156
+
| 5. | HDInsightHBaseLogs |**Head node**: HDFSGarbageCollectorLog, HDFSNameNodeLog **Worker node**: PhoenixServerLog, HBaseRegionServerLog, HBaseRestServerLog **Zookeeper node**: HBaseMasterLog | This table contains logs from HBase and its related components: Phoenix and HDFS. |
157
+
| 6. | HDInsightHBaseMetrics | No log types | This table contains JMX metrics from HBase. It contains all the same JMX metrics from the tables listed in the **Old Schema** column. In contrast from the old tables, each row contains one metric. |
158
158
| 7. | HDInsightHadoopAndYarn Metrics | No log types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
159
159
160
160
### Hadoop cluster
161
161
162
-
Different log types(sources) inside Hadoop tables
162
+
The following table names are for different log types(sources) inside Hadoop tables.
163
163
164
164
| Source number | Table name | Log types | Description |
165
165
| --- | --- | --- | --- |
@@ -169,7 +169,7 @@ Different log types(sources) inside Hadoop tables
169
169
| 4. | HDInsightHadoopAndYarnMetrics | No Log Types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
170
170
| 5. | HDInsightHiveAndLLAPLogs |**Head node**: HiveMetastoreLog, HiveServer2Log, WebHcatLog | This table contains logs generated from Hive, LLAP, and their related components: WebHCat and Zeppelin. |
171
171
| 6. | HDInsight Hive And LLAP Metrics | No log types | This table contains JMX metrics from the Hive and LLAP frameworks. It contains all the same JMX metrics as the old Custom Logs tables. It contains one metric per record. |
172
-
| 7. | HDInsight Security Logs |**Head node**: AmbariAuditLog, AuthLog **Zookeeper node**: AuthLog | This table contains records from the Ambari audit and authentication Logs. |
172
+
| 7. | HDInsight Security Logs |**Head node**: AmbariAuditLog, AuthLog **Zookeeper node**: AuthLog | This table contains records from the Ambari audit and authentication logs. |
173
173
174
174
## Parameter syntax
175
175
@@ -184,9 +184,7 @@ A parameter contains three parts:
184
184
185
185
### Syntax for multiple tables
186
186
187
-
When you have multiple tables, they're separated with a comma.
188
-
189
-
For example:
187
+
When you have multiple tables, they're separated with a comma. For example:
When you have multiple source types or log types, they're separated with a space.
198
196
199
197
To disable a source, write the table name that contains the log types, followed by a colon and then the real log type name:
200
-
`TableName : LogTypeName`
198
+
199
+
`TableName : LogTypeName`
201
200
202
201
For example, assume that `spark HDInsightSecurityLogs` is a table that has two log types: `AmbariAuditLog` and `AuthLog`. To disable both the log types, the correct syntax would be:
0 commit comments