Skip to content

Commit f0aa5de

Browse files
jborsecnikjborsecnik
authored
Merge pull request #189390 from jonburchel/2022-02-22-add-shir-monitoring-docs
Initial draft for new SHIR monitoring docs
2 parents 9f086c8 + f285c6e commit f0aa5de

File tree

5 files changed

+238
-0
lines changed

5 files changed

+238
-0
lines changed

‎articles/data-factory/TOC.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -835,6 +835,10 @@ items:
835835
- name: Diagnostic tool for self-hosted integration runtime
836836
href: self-hosted-integration-runtime-diagnostic-tool.md
837837
displayName: self hosted
838+
- name: Monitoring self-hosted intgegration runtime in Azure
839+
href: monitor-shir-in-azure.md
840+
- name: Configuring self-hosted integration runtime for log analytics collection
841+
href: how-to-configure-shir-for-log-analytics-collection.md
838842
displayName: self hosted, runtime
839843
- name: Azure-SSIS integration runtime
840844
items:

‎articles/data-factory/how-to-configure-shir-for-log-analytics-collection.md

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
---
2+
title: Configure the self-hosted integration runtime for log analytics collection
3+
titleSuffix: Azure Data Factory & Azure Synapse
4+
description: This article describes how to instrument the self-hosted integration runtime for log analytics collection.
5+
author: jonburchel
6+
ms.service: data-factory
7+
ms.subservice:
8+
ms.custom: synapse
9+
ms.topic: how-to
10+
ms.date: 02/22/2022
11+
ms.author: jburchel
12+
---
13+
14+
# Configure the self-hosted integration runtime (SHIR) for log analytics collection
15+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
16+
17+
## Prerequisites
18+
19+
An available Log Analytics workspace is required for this approach. We recommended you note down the workspace ID and authentication Key of your Log Analytics workspace as you might need it for certain scenarios. This solution will increase the data that will be sent to the Log Analytics workspace and will have a small impact on overall cost. Read on for details on how to keep the amount of data to a minimum.
20+
21+
## Objectives and scenarios
22+
23+
Centralize the events and the performance counter data to your Log Analytics workspace, first the virtual machine hosting the SHIR must be appropriately instrumented. Choose between two main scenarios below.
24+
25+
### Instrumenting on-premises virtual machines
26+
27+
The article [Install Log Analytics agent on Windows computers](../azure-monitor/agents/agent-windows.md) describes how to install the client on a virtual machine typically hosted on-premises. This can be either a physical server or a virtual machine hosted on a customer managed hypervisor. As mentioned in the prerequisite section, when installing the Log Analytics agent, you'll have to provide the Log Analytics workspace ID and Workspace Key to finalize the connection.
28+
29+
### Instrument Azure virtual machines
30+
31+
The recommended approach to instrument an Azure virtual machine based SHIR is to use virtual machine insights as described in the article [Enable VM insights overview](../azure-monitor/vm/vminsights-enable-overview.md). There are multiple ways to configure the Log Analytics agent when the SHIR is hosted in an Azure virtual machine. All the options are described in the article [Log Analytics agent overview](../azure-monitor/agents/log-analytics-agent.md#installation-options).
32+
33+
## Configure event log and performance counter capture
34+
35+
This step will highlight how to configure both Event viewer logs and performance counters to be sent to Log Analytics. The steps described below are common regardless of how the agent was deployed.
36+
37+
### Select event viewer journals
38+
39+
First you must collect event viewer journals relevant to the SHIR as described in the article [Collect Windows event log data sources with Log Analytics agent in Azure Monitor](../azure-monitor/agents/data-sources-windows-events.md).
40+
41+
It's important to note that when choosing the event logs using the interface, it's normal that you won't see all journals that can possibly exist on a machine. So, the two journals that we need for SHIR monitoring won't show up in this list. If you type the journal name exactly as it appears on the local virtual machine, it will be captured and sent to your Log analytics workspace.
42+
43+
The event journal name we must configure are:
44+
45+
- Connectors – Integration Runtime
46+
- Integration Runtime
47+
48+
:::image type="content" source="media/how-to-configure-shir-for-log-analytics-collection/configure-journals-for-collection.png" alt-text="Screenshot of the selection of the SHIR relevant logs with errors and warnings checked.":::
49+
50+
> [!IMPORTANT]
51+
> Leaving the **Information** level checked will increase the volume of data significantly if you have many SHIR hosts deployed and a larger number of scans. We strongly suggest you keep only Error and Warning.
52+
53+
### Select Performance counters
54+
55+
In the same configuration pane, you can click on **Windows Performance Counters** to select individual performance counters to send up to log analytics.
56+
57+
> [!IMPORTANT]
58+
> Keep in mind that performance counters are, by their nature, a continuous data stream. Therefore, it’s crucial that you consider the impact of data collection on the total cost of your Azure Monitor/Log Analytics deployment. Unless an allowed data ingestion budget has been granted and a constant ingestion of data has been allowed and budgeted for, gathering performance counters should only be configured for a defined period to establish a performance baseline.
59+
60+
In the interface, when first configuring it, a suggested counter set will be recommended. Select those that apply to the type of performance analysis you want to perform. **%CPU** and **Available memory** are commonly monitored counters but others like **Network bandwidth consumption** can be useful in scenarios where the data volume is large, and bandwidth or execution time are constrained.
61+
62+
:::image type="content" source="media/how-to-configure-shir-for-log-analytics-collection/select-counters.png" alt-text="Screenshot of the counter selection interface in the Azure portal.":::
63+
64+
## View Events and Performance counter data in Log Analytics
65+
66+
Consult this tutorial on [How to query data in Log Analytics](../azure-monitor/logs/log-analytics-tutorial.md).
67+
The two tables where the telemetry is saved are called Perf and Event respectively. The following query will check the row count to see if we have data flowing in. This would confirm if the instrumentation described above is working.
68+
69+
### Sample KQL queries
70+
71+
#### Check row counts
72+
73+
```kusto
74+
(
75+
Event
76+
| extend TableName = "Event"
77+
| summarize count() by TableName
78+
)
79+
| union
80+
(
81+
Perf
82+
| extend TableName = "Perf"
83+
| summarize count() by TableName
84+
)
85+
```
86+
87+
#### Query events
88+
89+
##### Retrieve the first 10 event rows
90+
91+
```kusto
92+
Event
93+
| take 10
94+
```
95+
96+
##### Retrieve the event count by message severity
97+
98+
```kusto
99+
Event
100+
| summarize count() by EventLevelName
101+
```
102+
103+
##### Render a pie chart of count by message severity
104+
105+
```kusto
106+
Event
107+
| summarize count() by EventLevelName
108+
| render piechart
109+
```
110+
111+
##### Retrieve all errors with a particular text string
112+
113+
Here we are searching for all messages that have the word _disconnected_ in them.
114+
115+
```kusto
116+
Event
117+
| where RenderedDescription has "disconnected"
118+
```
119+
120+
##### Multi-table search for a keyword without knowing the schema
121+
122+
The search command is useful when one does not know which column the information is contained in. This query returns all rows from the specified tables that have at least one column that contains the search term. The word is _disconnected_ in this example.
123+
124+
```kusto
125+
search in (Perf, Event) "disconnected"
126+
```
127+
128+
##### Retrieve all events from one specific log journal
129+
130+
In this example we’re narrowing the query to the log journal called **Connectors – Integration Runtime**.
131+
132+
```kusto
133+
Event
134+
| where EventLog == "Connectors – Integration Runtime"
135+
```
136+
137+
##### Use timespans to restrict query results
138+
139+
This query uses the same query as above but restricts results to those occurring 2 days ago or more recently.
140+
141+
```kusto
142+
Event
143+
| where EventLog == "Connectors – Integration Runtime"
144+
and TimeGenerated >= ago(2d)
145+
```
146+
147+
#### Query performance counter data
148+
149+
##### Retrieve the first 10 performance counter readings
150+
151+
```kusto
152+
Perf
153+
| take 10
154+
```
155+
156+
##### Retrieve a specific counter with time constraints
157+
158+
```kusto
159+
Perf
160+
| where TimeGenerated >= ago(24h)
161+
and ObjectName == "Network Adapter"
162+
and InstanceName == "Mellanox ConnectX-4 Lx Virtual Ethernet Adapter"
163+
and CounterName == "Bytes Received/sec"
164+
```
165+
166+
Performance counters are hierarchical in nature, so be mindful to have enough _where_ predicates in your query to select only the specific counter you need.
167+
168+
##### Retrieve 95th percentile for a given counter binned by 30 minute slices of the last 24 hours
169+
170+
This example is all the counters for a specific network adapter.
171+
172+
```kusto
173+
Perf
174+
| where TimeGenerated >= ago(24h)
175+
and ObjectName == "Network Adapter"
176+
and InstanceName == "Mellanox ConnectX-4 Lx Virtual Ethernet Adapter"
177+
| project TimeGenerated, Computer, ObjectName, InstanceName, CounterName, CounterValue
178+
| summarize percentile(CounterValue, 95) by bin(TimeGenerated, 30m), Computer, ObjectName, InstanceName, CounterName
179+
```
180+
181+
##### Use variables for code reusability
182+
183+
Here we are making the object name and counter name a variable so we do not have to change the KQL query body to make changes to those selections later.
184+
185+
```kusto
186+
let pObjectName = "Memory"; // Required to select the right counter
187+
let pCounterName = "Available MBytes"; // Required to select the right counter
188+
Perf
189+
| where Type == "Perf" and ObjectName == pObjectName and CounterName == pCounterName
190+
| project TimeGenerated, Computer, CounterName, CounterValue
191+
| order by TimeGenerated asc
192+
| summarize Value=max(CounterValue) by CounterName, TimeStamps=TimeGenerated
193+
```
194+
195+
## Next Steps
196+
197+
- [Review integration runtime concepts in Azure Data Factory.](concepts-integration-runtime.md)
198+
- Learn how to [create a self-hosted integration runtime in the Azure portal.](create-self-hosted-integration-runtime.md)
199+

‎articles/data-factory/media/how-to-configure-shir-for-log-analytics-collection/configure-journals-for-collection.png

53.5 KB
Loading

‎articles/data-factory/media/how-to-configure-shir-for-log-analytics-collection/select-counters.png

79.5 KB
Loading

‎articles/data-factory/monitor-shir-in-azure.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
title: Monitor self-hosted integration runtime (SHIR) virtual machines in Azure
3+
titleSuffix: Azure Data Factory & Azure Synapse
4+
description: This article describes how to monitor Azure virtual machines hosting the self-hosted integration runtime.
5+
author: jonburchel
6+
ms.service: data-factory
7+
ms.subservice:
8+
ms.custom: synapse
9+
ms.topic: conceptual
10+
ms.date: 02/22/2022
11+
ms.author: jburchel
12+
---
13+
14+
# Monitor self-hosted integration runtime (SHIR) virtual machines in Azure
15+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
16+
17+
By default, the Self Hosted Integration Runtime’s diagnostic and performance telemetry is saved locally on the virtual or physical machine that is hosting it. Two broad categories of telemetry are of interest for monitoring the Self Hosted Integration Runtime.
18+
19+
## Event logs
20+
21+
When logged on locally to the Self Hosted Integration Runtime, specific events can be viewed using the [event viewer](/windows/win32/eventlog/viewing-the-event-log.md). The relevant events are captured in two event viewer journals named: **Connectors – Integration Runtime** and **Integration Runtime** respectively. While it’s possible to log on to to the Self Hosted Integration Runtime hosts individually to view these events, it's also possible to stream these events to a Log Analytics workspace in Azure monitor for ease of query and centralization purposes.
22+
23+
## Performance counters
24+
25+
Performance counters in Windows and Linux provide insight into the performance of hardware components, operating systems, and applications such as the Self Hosted Integration Runtime. The performance counters can be viewed and collected locally on the VM using the performance monitor tool. See the article on [using performance counters](/windows/win32/perfctrs/using-performance-counters.md) for more details.
26+
27+
## Centralize log collection and analysis
28+
29+
When a deployment requires a more in-depth level of analysis or has reached a certain scale, it becomes impractical to log on to locally to each Self Hosted Integration Runtime host. Therefore, we recommend using Azure Monitor and Azure Log Analytics specifically to collect that data and enable a single pane of glass monitoring for your Self Hosted Integration Runtimes. See the article on [Configuring the SHIR for log analytics collection](how-to-configure-shir-for-log-analytics-collection.md) for instructions on how to instrument your Self Hosted Integration Runtimes for Azure Monitor.
30+
31+
## Next Steps
32+
33+
- [How to configure SHIR for log analytics collection](how-to-configure-shir-for-log-analytics-collection.md)
34+
- [Review integration runtime concepts in Azure Data Factory.](concepts-integration-runtime.md)
35+
- Learn how to [create a self-hosted integration runtime in the Azure portal.](create-self-hosted-integration-runtime.md)

0 commit comments

Comments
 (0)