You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-monitor/logs/search-jobs.md
+78-41Lines changed: 78 additions & 41 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,18 +18,18 @@ Search jobs are asynchronous queries that fetch records into a new search table
18
18
19
19
| Action | Permissions required |
20
20
|:-------|:---------------------|
21
-
|Run a search job|`Microsoft.OperationalInsights/workspaces/tables/write` and `Microsoft.OperationalInsights/workspaces/searchJobs/write` permissions to the Log Analytics workspace, for example, as provided by the [Log Analytics Contributor built-in role](../logs/manage-access.md#built-in-roles).|
21
+
|Run a search job|`Microsoft.OperationalInsights/workspaces/tables/write` and `Microsoft.OperationalInsights/workspaces/searchJobs/write` permissions to the Log Analytics workspace, for example, as provided by the [Log Analytics Contributor built-in role](../logs/manage-access.md#built-in-roles).|
22
22
23
23
## When to use search jobs
24
24
25
25
Use a search job when the log query timeout of 10 minutes isn't sufficient to search through large volumes of data or if you're running a slow query.
26
26
27
27
Search jobs also let you retrieve records from [Archived Logs](data-retention-archive.md) and [Basic Logs](basic-logs-configure.md) tables into a new log table you can use for queries. In this way, running a search job can be an alternative to:
28
28
29
-
-[Restoring data from Archived Logs](restore.md) for a specific time range.<br/>
29
+
*[Restoring data from Archived Logs](restore.md) for a specific time range.<br/>
30
30
Use restore when you have a temporary need to run many queries on a large volume of data.
31
31
32
-
- Querying Basic Logs directly and paying for each query.<br/>
32
+
* Querying Basic Logs directly and paying for each query.<br/>
33
33
To determine which alternative is more cost-effective, compare the cost of querying Basic Logs with the cost of running a search job and storing the search job results.
34
34
35
35
## What does a search job do?
@@ -40,12 +40,12 @@ The search job results table is an [Analytics table](../logs/basic-logs-configur
40
40
41
41
The search results table schema is based on the source table schema and the specified query. The following other columns help you track the source records:
42
42
43
-
| Column | Value |
44
-
|:---|:---|
45
-
|_OriginalType |*Type* value from source table. |
46
-
|_OriginalItemId |*_ItemID* value from source table. |
|_OriginalItemId |*_ItemID* value from source table. |
47
47
|_OriginalTimeGenerated |*TimeGenerated* value from source table. |
48
-
| TimeGenerated | Time at which the search job ran. |
48
+
| TimeGenerated | Time at which the search job ran. |
49
49
50
50
Queries on the results table appear in [log query auditing](query-audit.md) but not the initial search job.
51
51
@@ -61,13 +61,15 @@ Run a search job to fetch records from large datasets into a new search results
61
61
To run a search job, in the Azure portal:
62
62
63
63
1. From the **Log Analytics workspace** menu, select **Logs**.
64
+
64
65
1. Select the ellipsis menu on the right-hand side of the screen and toggle **Search job mode** on.
65
66
66
67
:::image type="content" source="media/search-job/switch-to-search-job-mode.png" alt-text="Screenshot of the Logs screen with the Search job mode switch highlighted." lightbox="media/search-job/switch-to-search-job-mode.png":::
67
68
68
69
Azure Monitor Logs intellisense supports [KQL query limitations in search job mode](#kql-query-limitations) to help you write your search job query.
69
70
70
71
1. Specify the search job date range using the time picker.
72
+
71
73
1. Type the search job query and select the **Search Job** button.
72
74
73
75
Azure Monitor Logs prompts you to provide a name for the result set table and informs you that the search job is subject to billing.
@@ -93,6 +95,7 @@ To run a search job, in the Azure portal:
93
95
:::image type="content" source="media/search-job/search-job-done.png" alt-text="Screenshot that shows an Azure Monitor Logs message that the search job is done." lightbox="media/search-job/search-job-done.png":::
94
96
95
97
### [API](#tab/api-1)
98
+
96
99
To run a search job, call the **Tables - Create or Update** API. The call includes the name of the results table to be created. The name of the results table must end with *_SRCH*.
97
100
98
101
```http
@@ -103,13 +106,12 @@ PUT https://management.azure.com/subscriptions/{subscriptionId}/resourcegroups/{
103
106
104
107
Include the following values in the body of the request:
105
108
106
-
|Name | Type | Description |
107
-
| --- | --- | --- |
108
-
|properties.searchResults.query | string | Log query written in KQL to retrieve data. |
109
-
|properties.searchResults.limit | integer | Maximum number of records in the result set, up to one million records. (Optional)|
110
-
|properties.searchResults.startSearchTime | string |Start of the time range to search. |
111
-
|properties.searchResults.endSearchTime | string | End of the time range to search. |
| properties.searchResults.query | string | Log query written in KQL to retrieve data. |
112
+
| properties.searchResults.limit | integer | Maximum number of records in the result set, up to one million records. (Optional) |
113
+
| properties.searchResults.startSearchTime | string | Start of the time range to search. |
114
+
| properties.searchResults.endSearchTime | string | End of the time range to search. |
113
115
114
116
**Sample request**
115
117
@@ -150,11 +152,34 @@ For example:
150
152
az monitor log-analytics workspace table search-job create --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name HeartbeatByIp_SRCH --search-query 'Heartbeat | where ComputerIP has "00.000.00.000"' --limit 1500 --start-search-time "2022-01-01T00:00:00.000Z" --end-search-time "2022-01-08T00:00:00.000Z" --no-wait
151
153
```
152
154
155
+
### [PowerShell](#tab/powershell-1)
156
+
157
+
```powershell
158
+
New-AzOperationalInsightsSearchTable
159
+
[-ResourceGroupName] <String>
160
+
[-WorkspaceName] <String>
161
+
[-TableName] <String>
162
+
[[-RetentionInDays] <Int32>]
163
+
[-TotalRetentionInDays <Int32>]
164
+
-SearchQuery <String>
165
+
-StartSearchTime <String>
166
+
-EndSearchTime <String>
167
+
[-Limit <Int32>]
168
+
[-AsJob]
169
+
[-DefaultProfile <IAzureContextContainer>]
170
+
[-WhatIf]
171
+
[-Confirm]
172
+
[<CommonParameters>]
173
+
```
174
+
153
175
---
154
176
155
177
## Get search job status and details
178
+
156
179
### [Portal](#tab/portal-2)
157
-
1. From the **Log Analytics workspace** menu, select **Logs**.
180
+
181
+
1. From the **Log Analytics workspace** menu, select **Logs**.
182
+
158
183
1. From the Tables tab, select **Search results** to view all search job results tables.
159
184
160
185
The icon on the search job results table displays an update indication until the search job is completed.
@@ -168,17 +193,16 @@ Call the **Tables - Get** API to get the status and details of a search job:
168
193
GET https://management.azure.com/subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/tables/<TableName>_SRCH?api-version=2021-12-01-preview
169
194
```
170
195
171
-
**Table status**<br>
196
+
**Table status**
172
197
173
198
Each search job table has a property called *provisioningState*, which can have one of the following values:
174
199
175
-
| Status | Description |
176
-
|:---|:---|
177
-
| Updating | Populating the table and its schema. |
| Updating | Populating the table and its schema.|
178
203
| InProgress | Search job is running, fetching data. |
179
-
| Succeeded | Search job completed. |
180
-
| Deleting | Deleting the search job table. |
181
-
204
+
| Succeeded | Search job completed. |
205
+
| Deleting | Deleting the search job table. |
182
206
183
207
**Sample request**
184
208
@@ -234,45 +258,58 @@ For example:
234
258
az monitor log-analytics workspace table show --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name HeartbeatByIp_SRCH --output table \
235
259
```
236
260
261
+
### [PowerShell](#tab/powershell-2)
262
+
263
+
```powershell
264
+
Get-AzOperationalInsightsTable
265
+
[-ResourceGroupName] <String>
266
+
[-WorkspaceName] <String>
267
+
[[-TableName] <String>]
268
+
[-DefaultProfile <IAzureContextContainer>]
269
+
[<CommonParameters>]
270
+
```
271
+
237
272
---
238
273
239
274
## Delete a search job table
275
+
240
276
We recommend you [delete the search job table](../logs/create-custom-table.md#delete-a-table) when you're done querying the table. This reduces workspace clutter and extra charges for data retention.
241
277
242
278
## Limitations
279
+
243
280
Search jobs are subject to the following limitations:
244
281
245
-
- Optimized to query one table at a time.
246
-
- Search date range is up to one year.
247
-
- Supports long running searches up to a 24-hour time-out.
248
-
- Results are limited to one million records in the record set.
249
-
- Concurrent execution is limited to five search jobs per workspace.
250
-
- Limited to 100 search results tables per workspace.
251
-
- Limited to 100 search job executions per day per workspace.
282
+
* Optimized to query one table at a time.
283
+
* Search date range is up to one year.
284
+
* Supports long running searches up to a 24-hour time-out.
285
+
* Results are limited to one million records in the record set.
286
+
* Concurrent execution is limited to five search jobs per workspace.
287
+
* Limited to 100 search results tables per workspace.
288
+
* Limited to 100 search job executions per day per workspace.
252
289
253
290
When you reach the record limit, Azure aborts the job with a status of *partial success*, and the table will contain only records ingested up to that point.
254
291
255
292
### KQL query limitations
256
293
257
294
Search jobs are intended to scan large volumes of data in a specific table. Therefore, search job queries must always start with a table name. To enable asynchronous execution using distribution and segmentation, the query supports a subset of KQL, including the operators:
0 commit comments