Skip to content

Commit c447784

Browse files
PMEds28PMEds28
authored
Merge pull request #276204 from cdpark/horz-monitor-hdinsight
User Story 246543: Q&M: Curate 27 SOX Azure Monitor horizontals articles to reduce duplication - HD Insight
2 parents 314ca4d + aeaad19 commit c447784

7 files changed

+17
-69
lines changed

‎articles/hdinsight/hdinsight-autoscale-clusters.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,9 @@ Autoscale continuously monitors the cluster and collects the following metrics:
4141
|Used Memory per Node|The load on a worker node. A worker node on which 10 GB of memory is used, is considered under more load than a worker with 2 GB of used memory.|
4242
|Number of Application Masters per Node|The number of Application Master (AM) containers running on a worker node. A worker node that is hosting two AM containers, is considered more important than a worker node that is hosting zero AM containers.|
4343

44-
The above metrics are checked every 60 seconds. Autoscale makes scale-up and scale-down decisions based on these metrics.
44+
The above metrics are checked every 60 seconds. Autoscale makes scale-up and scale-down decisions based on these metrics.
45+
46+
For a complete list of cluster metrics, see [Supported metrics for Microsoft.HDInsight/clusters](monitor-hdinsight-reference.md#supported-metrics-for-microsofthdinsightclusters).
4547

4648
### Load-based scale conditions
4749

‎articles/hdinsight/hdinsight-hadoop-oms-log-analytics-tutorial.md

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,10 @@ HDInsight support cluster auditing with Azure Monitor logs, by importing the fol
153153
* `log_gateway_audit_CL` - this table provides audit logs from cluster gateway nodes that show successful and failed sign-in attempts.
154154
* `log_auth_CL` - this table provides SSH logs with successful and failed sign-in attempts.
155155
* `log_ambari_audit_CL` - this table provides audit logs from Ambari.
156-
* `log_ranger_audti_CL` - this table provides audit logs from Apache Ranger on ESP clusters.
156+
* `ranger_audit_logs_CL` - this table provides audit logs from Apache Ranger on ESP clusters.
157+
158+
For the log table mappings from the classic Azure Monitor integration to the new one, see [Log table mapping](monitor-hdinsight-reference.md#log-table-mapping).
159+
157160

158161
#### [Classic Azure Monitor experience](#tab/previous)
159162

@@ -303,7 +306,9 @@ HDInsight support cluster auditing with Azure Monitor logs, by importing the fol
303306
* `log_gateway_audit_CL` - this table provides audit logs from cluster gateway nodes that show successful and failed sign-in attempts.
304307
* `log_auth_CL` - this table provides SSH logs with successful and failed sign-in attempts.
305308
* `log_ambari_audit_CL` - this table provides audit logs from Ambari.
306-
* `log_ranger_audti_CL` - this table provides audit logs from Apache Ranger on ESP clusters.
309+
* `ranger_audit_logs_CL` - this table provides audit logs from Apache Ranger on ESP clusters.
310+
311+
For the log table mappings from the classic Azure Monitor integration to the new one, see [Log table mapping](monitor-hdinsight-reference.md#log-table-mapping).
307312

308313
---
309314

‎articles/hdinsight/kafka/apache-kafka-log-analytics-operations-management.md

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -84,13 +84,7 @@ The steps to enable Azure Monitor logs for HDInsight are the same for all HDInsi
8484
| summarize AggregatedValue = avg(kafka_BrokerTopicMetrics_BytesOutPerSec_Count_value_d) by bin(TimeGenerated, 1h)
8585
```
8686
87-
You can also enter `*` to search all types logged. Currently the following logs are available for queries:
88-
89-
| Log type | Description |
90-
| ---- | ---- |
91-
| log\_kafkaserver\_CL | Kafka broker server.log |
92-
| log\_kafkacontroller\_CL | Kafka broker controller.log |
93-
| metrics\_kafka\_CL | Kafka JMX metrics |
87+
You can also enter `*` to search all types logged. For a list of logs that are available for queries, see [Kafka workload](../monitor-hdinsight-reference.md#kafka-workload).
9488
9589
:::image type="content" source="./media/apache-kafka-log-analytics-operations-management/apache-kafka-cpu-usage.png" alt-text="Apache kafka log analytics cpu usage." border="true":::
9690

‎articles/hdinsight/log-analytics-migration.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,15 +133,15 @@ To enable the new Azure Monitor integration, follow the steps outlined in the [A
133133

134134
Since the new table format is different from the previous one, your queries need to be reworked so you can use our new tables. Once you enable the new Azure Monitor integration, you can browse the tables and schemas to identify the fields that are used in your old queries.
135135

136-
We provide a [mapping table](#appendix-table-mapping) between the old table to the new table to help you quickly find the new fields you need to use to migrate your dashboards and queries.
136+
We provide a [mapping table](monitor-hdinsight-reference.md#log-table-mapping) between the old table to the new table to help you quickly find the new fields you need to use to migrate your dashboards and queries.
137137

138138
**Default queries**: We created default queries that show how to use the new tables for common situations. The default queries also show what information is available in each table. You can access the default queries by following the instructions in the [Default queries to use with new tables](#default-queries-to-use-with-new-tables) section in this article.
139139

140140
### Update dashboards for HDInsight clusters
141141

142142
If you have built multiple dashboards to monitor your HDInsight clusters, you need to adjust the query behind the table once you enable the new Azure Monitor integration. The table name or the field name might change in the new integration, but all the information you have in old integration is included.
143143

144-
Refer to the [mapping table](#appendix-table-mapping) between the old table/schema to the new table/schema to update the query behind the dashboards.
144+
Refer to the [mapping table](monitor-hdinsight-reference.md#log-table-mapping) between the old table/schema to the new table/schema to update the query behind the dashboards.
145145

146146
#### Out-of-box dashboards
147147

‎articles/hdinsight/monitor-hdinsight-reference.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ See [Monitor HDInsight](monitor-hdinsight.md) for details on the data you can co
1818
### Supported metrics for Microsoft.HDInsight/clusters
1919
The following table lists the metrics available for the Microsoft.HDInsight/clusters resource type.
2020
[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
21-
[!INCLUDE [Microsoft.HDInsight/clusters](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-hdinsight-clusters-metrics-include.md)]
21+
[!INCLUDE [microsoft-hdinsight-clusters-metrics-include](~/reusable-content/ce-skilling/azure/includes/azure-monitor/reference/metrics/microsoft-hdinsight-clusters-metrics-include.md)]
2222

2323
[!INCLUDE [horz-monitor-ref-metrics-dimensions-intro](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-dimensions-intro.md)]
2424

‎articles/hdinsight/monitor-hdinsight.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,8 @@ The following table describes a couple of alert rules for HDInsight. These alert
134134
| Metric| Pending CPU | Whenever the maximum pending CPU is greater or less than dynamic threshold|
135135
| Activity log| Delete cluster | Whenever the Activity Log has an event with Category='Administrative', Signal name='Delete Cluster (HDInsight Cluster)'|
136136

137+
For an example that shows how to create an alert, see [Azure Monitor alerts](cluster-availability-monitor-logs.md#azure-monitor-alerts).
138+
137139
[!INCLUDE [horz-monitor-advisor-recommendations](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-advisor-recommendations.md)]
138140

139141
## Related content

‎articles/hdinsight/selective-logging-analysis.md

Lines changed: 1 addition & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -114,62 +114,7 @@ If the script action shows a failure status in the script action history:
114114

115115
## Table names
116116

117-
### Spark cluster
118-
119-
The following table names are for different log types (sources) inside Spark tables.
120-
121-
| Source number | Table name | Log types | Description |
122-
| --- | --- | --- | --- |
123-
| 1. | HDInsightAmbariCluster Alerts | No log types | This table contains Ambari cluster alerts from each node in the cluster (except for edge nodes). Each alert is a record in this table.
124-
| 2. | HDInsightAmbariSystem Metrics | No log types | This table contains system metrics collected from Ambari. The metrics now come from each node in the cluster (except for edge nodes) instead of just the two head nodes. Each metric is now a column, and each metric is reported once per record. |
125-
| 3. | HDInsightHadoopAnd YarnLogs | **Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node**: NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
126-
| 4. | HDInsightSecurityLogs | AmbariAuditLog, AuthLog | This table contains records from the Ambari audit and authentication logs. |
127-
| 5. | HDInsightSparkLogs | **Head node**: JupyterLog, LivyLog, SparkThriftDriverLog **Worker node**: SparkExecutorLog, SparkDriverLog | This table contains all logs related to Spark and its related components: Livy and Jupyter. |
128-
| 6. | HDInsightHadoopAnd YarnMetrics | No log types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
129-
| 7. | HDInsightOozieLogs | Oozie | This table contains all logs generated from the Oozie framework. |
130-
131-
### Interactive Query cluster
132-
133-
The following table names are for different log types (sources) inside Interactive Query tables.
134-
135-
| Source number | Table name | Log types | Description |
136-
| --- | --- | --- | --- |
137-
| 1. | HDInsightAmbariClusterAlerts | No log types | This table contains Ambari cluster alerts from each node in the cluster (except for edge nodes). Each alert is a record in this table. |
138-
| 2. | HDInsightAmbariSystem Metrics | No log types | This table contains system metrics collected from Ambari. The metrics now come from each node in the cluster (except for edge nodes) instead of just the two head nodes. Each metric is now a column, and each metric is reported once per record. |
139-
| 3. | HDInsightHadoopAndYarnLogs | **Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node**: NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
140-
| 4. | HDInsightHadoopAndYarnMetrics | No log types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
141-
| 5. | HDInsightHiveAndLLAPLogs | **Head node**: InteractiveHiveHSILog, InteractiveHiveMetastoreLog, ZeppelinLog | This table contains logs generated from Hive, LLAP, and their related components: WebHCat and Zeppelin. |
142-
| 6. | HDInsightHiveAndLLAPmetrics | No log types | This table contains JMX metrics from the Hive and LLAP frameworks. It contains all the same JMX metrics as the old Custom Logs tables. It contains one metric per record. |
143-
| 7. | HDInsightHiveTezAppStats | No log types |
144-
| 8. | HDInsightSecurityLogs | **Head node**: AmbariAuditLog, AuthLog **ZooKeeper node, worker node**: AuthLog | This table contains records from the Ambari audit and authentication logs. |
145-
146-
### HBase cluster
147-
148-
The following table names are for different log types (sources) inside HBase tables.
149-
150-
| Source number | Table name | Log types | Description |
151-
| --- | --- | --- | --- |
152-
| 1. | HDInsightAmbariClusterAlerts | No other log types | This table contains Ambari cluster alerts from each node in the cluster (except for edge nodes). Each alert is a record in this table.
153-
| 2. | HDInsightAmbariSystem Metrics | No other log types | This table contains system metrics collected from Ambari. The metrics now come from each node in the cluster (except for edge nodes) instead of just the two head nodes. Each metric is now a column, and each metric is reported once per record. |
154-
| 3. | HDInsightHadoopAndYarnLogs | **Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node**: NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
155-
| 4. | HDInsightSecurityLogs | **Head node**: AmbariAuditLog, AuthLog **Worker node**: AuthLog **ZooKeeper node**: AuthLog | This table contains records from the Ambari audit and authentication logs. |
156-
| 5. | HDInsightHBaseLogs | **Head node**: HDFSGarbageCollectorLog, HDFSNameNodeLog **Worker node**: PhoenixServerLog, HBaseRegionServerLog, HBaseRestServerLog **ZooKeeper node**: HBaseMasterLog | This table contains logs from HBase and its related components: Phoenix and HDFS. |
157-
| 6. | HDInsightHBaseMetrics | No log types | This table contains JMX metrics from HBase. It contains all the same JMX metrics from the tables listed in the Old Schema column. In contrast from the old tables, each row contains one metric. |
158-
| 7. | HDInsightHadoopAndYarn Metrics | No log types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
159-
160-
### Hadoop cluster
161-
162-
The following table names are for different log types (sources) inside Hadoop tables.
163-
164-
| Source number | Table name | Log types | Description |
165-
| --- | --- | --- | --- |
166-
| 1. | HDInsightAmbariClusterAlerts | No log types | This table contains Ambari cluster alerts from each node in the cluster (except for edge nodes). Each alert is a record in this table. |
167-
| 2. | HDInsightAmbariSystem Metrics | No log types | This table contains system metrics collected from Ambari. The metrics now come from each node in the cluster (except for edge nodes) instead of just the two head nodes. Each metric is now a column, and each metric is reported once per record. |
168-
| 3. | HDInsightHadoopAndYarnLogs | **Head node**: MRJobSummary, Resource Manager, TimelineServer **Worker node**: NodeManager | This table contains all logs generated from the Hadoop and YARN frameworks. |
169-
| 4. | HDInsightHadoopAndYarnMetrics | No log types | This table contains JMX metrics from the Hadoop and YARN frameworks. It contains all the same JMX metrics as the old Custom Logs tables, plus more metrics that we considered important. We added Timeline Server, Node Manager, and Job History Server metrics. It contains one metric per record. |
170-
| 5. | HDInsightHiveAndLLAPLogs | **Head node**: HiveMetastoreLog, HiveServer2Log, WebHcatLog | This table contains logs generated from Hive, LLAP, and their related components: WebHCat and Zeppelin. |
171-
| 6. | HDInsight Hive And LLAP Metrics | No log types | This table contains JMX metrics from the Hive and LLAP frameworks. It contains all the same JMX metrics as the old Custom Logs tables. It contains one metric per record. |
172-
| 7. | HDInsight Security Logs | **Head node**: AmbariAuditLog, AuthLog **ZooKeeper node**: AuthLog | This table contains records from the Ambari audit and authentication logs. |
117+
For a complete listing of table names for different log types (sources), see [Azure Monitor Logs tables](monitor-hdinsight-reference.md#azure-monitor-logs-tables).
173118

174119
## Parameter syntax
175120

0 commit comments

Comments
 (0)