Skip to content

Commit e023d6d

Browse files
author
Jill Grant
authored
Merge pull request #280892 from kainawroth/kainawroth-searchjobs
Adding PowerShell cmdlets to Search Jobs article
2 parents dd98b8c + bb7daec commit e023d6d

File tree

1 file changed

+72
-43
lines changed

1 file changed

+72
-43
lines changed

‎articles/azure-monitor/logs/search-jobs.md

Lines changed: 72 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -18,18 +18,18 @@ Search jobs are asynchronous queries that fetch records into a new search table
1818

1919
| Action | Permissions required |
2020
|:-------|:---------------------|
21-
|Run a search job| `Microsoft.OperationalInsights/workspaces/tables/write` and `Microsoft.OperationalInsights/workspaces/searchJobs/write` permissions to the Log Analytics workspace, for example, as provided by the [Log Analytics Contributor built-in role](../logs/manage-access.md#built-in-roles).|
21+
| Run a search job | `Microsoft.OperationalInsights/workspaces/tables/write` and `Microsoft.OperationalInsights/workspaces/searchJobs/write` permissions to the Log Analytics workspace, for example, as provided by the [Log Analytics Contributor built-in role](../logs/manage-access.md#built-in-roles). |
2222

2323
## When to use search jobs
2424

2525
Use a search job when the log query timeout of 10 minutes isn't sufficient to search through large volumes of data or if you're running a slow query.
2626

2727
Search jobs also let you retrieve records from [Archived Logs](data-retention-archive.md) and [Basic Logs](basic-logs-configure.md) tables into a new log table you can use for queries. In this way, running a search job can be an alternative to:
2828

29-
- [Restoring data from Archived Logs](restore.md) for a specific time range.<br/>
29+
* [Restoring data from Archived Logs](restore.md) for a specific time range.<br/>
3030
Use restore when you have a temporary need to run many queries on a large volume of data.
3131

32-
- Querying Basic Logs directly and paying for each query.<br/>
32+
* Querying Basic Logs directly and paying for each query.<br/>
3333
To determine which alternative is more cost-effective, compare the cost of querying Basic Logs with the cost of running a search job and storing the search job results.
3434

3535
## What does a search job do?
@@ -40,12 +40,12 @@ The search job results table is an [Analytics table](../logs/basic-logs-configur
4040

4141
The search results table schema is based on the source table schema and the specified query. The following other columns help you track the source records:
4242

43-
| Column | Value |
44-
|:---|:---|
45-
| _OriginalType | *Type* value from source table. |
46-
| _OriginalItemId | *_ItemID* value from source table. |
43+
| Column | Value |
44+
|:-----------------------|:-----------------------------------------|
45+
| _OriginalType | *Type* value from source table. |
46+
| _OriginalItemId | *_ItemID* value from source table. |
4747
| _OriginalTimeGenerated | *TimeGenerated* value from source table. |
48-
| TimeGenerated | Time at which the search job ran. |
48+
| TimeGenerated | Time at which the search job ran. |
4949

5050
Queries on the results table appear in [log query auditing](query-audit.md) but not the initial search job.
5151

@@ -61,13 +61,15 @@ Run a search job to fetch records from large datasets into a new search results
6161
To run a search job, in the Azure portal:
6262

6363
1. From the **Log Analytics workspace** menu, select **Logs**.
64+
6465
1. Select the ellipsis menu on the right-hand side of the screen and toggle **Search job mode** on.
6566

6667
:::image type="content" source="media/search-job/switch-to-search-job-mode.png" alt-text="Screenshot of the Logs screen with the Search job mode switch highlighted." lightbox="media/search-job/switch-to-search-job-mode.png":::
6768

6869
Azure Monitor Logs intellisense supports [KQL query limitations in search job mode](#kql-query-limitations) to help you write your search job query.
6970

7071
1. Specify the search job date range using the time picker.
72+
7173
1. Type the search job query and select the **Search Job** button.
7274

7375
Azure Monitor Logs prompts you to provide a name for the result set table and informs you that the search job is subject to billing.
@@ -93,6 +95,7 @@ To run a search job, in the Azure portal:
9395
:::image type="content" source="media/search-job/search-job-done.png" alt-text="Screenshot that shows an Azure Monitor Logs message that the search job is done." lightbox="media/search-job/search-job-done.png":::
9496

9597
### [API](#tab/api-1)
98+
9699
To run a search job, call the **Tables - Create or Update** API. The call includes the name of the results table to be created. The name of the results table must end with *_SRCH*.
97100

98101
```http
@@ -103,13 +106,12 @@ PUT https://management.azure.com/subscriptions/{subscriptionId}/resourcegroups/{
103106

104107
Include the following values in the body of the request:
105108

106-
|Name | Type | Description |
107-
| --- | --- | --- |
108-
|properties.searchResults.query | string | Log query written in KQL to retrieve data. |
109-
|properties.searchResults.limit | integer | Maximum number of records in the result set, up to one million records. (Optional)|
110-
|properties.searchResults.startSearchTime | string |Start of the time range to search. |
111-
|properties.searchResults.endSearchTime | string | End of the time range to search. |
112-
109+
| Name | Type | Description |
110+
| ---- | ---- | ------------------------------------------------------------------------------------------------------------------------- |
111+
| properties.searchResults.query | string | Log query written in KQL to retrieve data. |
112+
| properties.searchResults.limit | integer | Maximum number of records in the result set, up to one million records. (Optional) |
113+
| properties.searchResults.startSearchTime | string | Start of the time range to search. |
114+
| properties.searchResults.endSearchTime | string | End of the time range to search. |
113115

114116
**Sample request**
115117

@@ -144,17 +146,30 @@ Status code: 202 accepted.
144146

145147
To run a search job, run the [az monitor log-analytics workspace table search-job create](/cli/azure/monitor/log-analytics/workspace/table/search-job#az-monitor-log-analytics-workspace-table-search-job-create) command. The name of the results table, which you set using the `--name` parameter, must end with *_SRCH*.
146148

147-
For example:
149+
**Example**
148150

149151
```azurecli
150152
az monitor log-analytics workspace table search-job create --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name HeartbeatByIp_SRCH --search-query 'Heartbeat | where ComputerIP has "00.000.00.000"' --limit 1500 --start-search-time "2022-01-01T00:00:00.000Z" --end-search-time "2022-01-08T00:00:00.000Z" --no-wait
151153
```
152154

155+
### [PowerShell](#tab/powershell-1)
156+
157+
To run a search job, run the [New-AzOperationalInsightsSearchTable](/powershell/module/az.operationalinsights/new-azoperationalinsightssearchtable) command. The name of the results table, which you set using the `TableName` parameter, must end with *_SRCH*.
158+
159+
**Example**
160+
161+
```powershell
162+
New-AzOperationalInsightsSearchTable -ResourceGroupName ContosoRG -WorkspaceName ContosoWorkspace -TableName HeartbeatByIp_SRCH -SearchQuery "Heartbeat" -StartSearchTime "01-01-2022 00:00:00" -EndSearchTime "01-01-2022 00:00:00"
163+
```
164+
153165
---
154166

155167
## Get search job status and details
168+
156169
### [Portal](#tab/portal-2)
157-
1. From the **Log Analytics workspace** menu, select **Logs**.
170+
171+
1. From the **Log Analytics workspace** menu, select **Logs**.
172+
158173
1. From the Tables tab, select **Search results** to view all search job results tables.
159174

160175
The icon on the search job results table displays an update indication until the search job is completed.
@@ -168,17 +183,16 @@ Call the **Tables - Get** API to get the status and details of a search job:
168183
GET https://management.azure.com/subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/tables/<TableName>_SRCH?api-version=2021-12-01-preview
169184
```
170185

171-
**Table status**<br>
186+
**Table status**
172187

173188
Each search job table has a property called *provisioningState*, which can have one of the following values:
174189

175-
| Status | Description |
176-
|:---|:---|
177-
| Updating | Populating the table and its schema. |
190+
| Status | Description |
191+
|:-----------|:--------------------------------------|
192+
| Updating | Populating the table and its schema. |
178193
| InProgress | Search job is running, fetching data. |
179-
| Succeeded | Search job completed. |
180-
| Deleting | Deleting the search job table. |
181-
194+
| Succeeded | Search job completed. |
195+
| Deleting | Deleting the search job table. |
182196

183197
**Sample request**
184198

@@ -228,51 +242,66 @@ GET https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000
228242

229243
To check the status and details of a search job table, run the [az monitor log-analytics workspace table show](/cli/azure/monitor/log-analytics/workspace/table#az-monitor-log-analytics-workspace-table-show) command.
230244

231-
For example:
245+
**Example**
232246

233247
```azurecli
234248
az monitor log-analytics workspace table show --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name HeartbeatByIp_SRCH --output table \
235249
```
236250

251+
### [PowerShell](#tab/powershell-2)
252+
253+
To check the status and details of a search job table, run the [Get-AzOperationalInsightsTable](/powershell/module/az.operationalinsights/get-azoperationalinsightstable) command.
254+
255+
**Example**
256+
257+
```powershell
258+
Get-AzOperationalInsightsTable -ResourceGroupName "ContosoRG" -WorkspaceName "ContosoWorkspace" -tableName "HeartbeatByIp_SRCH"
259+
```
260+
261+
> [!NOTE]
262+
> When "-TableName" is not provided, the command will instead list all tables associated with a workspace.
263+
237264
---
238265

239266
## Delete a search job table
267+
240268
We recommend you [delete the search job table](../logs/create-custom-table.md#delete-a-table) when you're done querying the table. This reduces workspace clutter and extra charges for data retention.
241269

242270
## Limitations
271+
243272
Search jobs are subject to the following limitations:
244273

245-
- Optimized to query one table at a time.
246-
- Search date range is up to one year.
247-
- Supports long running searches up to a 24-hour time-out.
248-
- Results are limited to one million records in the record set.
249-
- Concurrent execution is limited to five search jobs per workspace.
250-
- Limited to 100 search results tables per workspace.
251-
- Limited to 100 search job executions per day per workspace.
274+
* Optimized to query one table at a time.
275+
* Search date range is up to one year.
276+
* Supports long running searches up to a 24-hour time-out.
277+
* Results are limited to one million records in the record set.
278+
* Concurrent execution is limited to five search jobs per workspace.
279+
* Limited to 100 search results tables per workspace.
280+
* Limited to 100 search job executions per day per workspace.
252281

253282
When you reach the record limit, Azure aborts the job with a status of *partial success*, and the table will contain only records ingested up to that point.
254283

255284
### KQL query limitations
256285

257286
Search jobs are intended to scan large volumes of data in a specific table. Therefore, search job queries must always start with a table name. To enable asynchronous execution using distribution and segmentation, the query supports a subset of KQL, including the operators:
258287

259-
- [where](/azure/data-explorer/kusto/query/whereoperator)
260-
- [extend](/azure/data-explorer/kusto/query/extendoperator)
261-
- [project](/azure/data-explorer/kusto/query/projectoperator)
262-
- [project-away](/azure/data-explorer/kusto/query/projectawayoperator)
263-
- [project-keep](/azure/data-explorer/kusto/query/project-keep-operator)
264-
- [project-rename](/azure/data-explorer/kusto/query/projectrenameoperator)
265-
- [project-reorder](/azure/data-explorer/kusto/query/projectreorderoperator)
266-
- [parse](/azure/data-explorer/kusto/query/parse-operator)
267-
- [parse-where](/azure/data-explorer/kusto/query/parse-where-operator)
288+
* [where](/azure/data-explorer/kusto/query/whereoperator)
289+
* [extend](/azure/data-explorer/kusto/query/extendoperator)
290+
* [project](/azure/data-explorer/kusto/query/projectoperator)
291+
* [project-away](/azure/data-explorer/kusto/query/projectawayoperator)
292+
* [project-keep](/azure/data-explorer/kusto/query/project-keep-operator)
293+
* [project-rename](/azure/data-explorer/kusto/query/projectrenameoperator)
294+
* [project-reorder](/azure/data-explorer/kusto/query/projectreorderoperator)
295+
* [parse](/azure/data-explorer/kusto/query/parse-operator)
296+
* [parse-where](/azure/data-explorer/kusto/query/parse-where-operator)
268297

269298
You can use all functions and binary operators within these operators.
270299

271300
## Pricing model
272301
The charge for a search job is based on:
273302

274-
- Search job execution - the amount of data the search job scans.
275-
- Search job results - the amount of data the search job finds and is ingested into the results table, based on the regular log data ingestion prices.
303+
* Search job execution - the amount of data the search job scans.
304+
* Search job results - the amount of data the search job finds and is ingested into the results table, based on the regular log data ingestion prices.
276305

277306
For example, if your table holds 500 GB per day, for a search over 30 days, you'll be charged for 15,000 GB of scanned data.
278307
If the search job finds 1,000 records that match the search query, you'll be charged for ingesting these 1,000 records into the results table.

0 commit comments

Comments
 (0)