|
| 1 | +--- |
| 2 | +title: Transform text logs during ingestion in Azure Monitor Logs |
| 3 | +description: Write a KQL query that transforms text log data and add the transformation to a data collection rule in Azure Monitor Logs. |
| 4 | +ms.topic: conceptual |
| 5 | +ms.date: 01/23/2023 |
| 6 | +author: guywi-ms |
| 7 | +ms.author: guywild |
| 8 | +ms.reviewer: jeffwo |
| 9 | +--- |
| 10 | +# Tutorial: Transform text logs during ingestion in Azure Monitor Logs |
| 11 | + |
| 12 | +Ingestion-time transformations let you filter or modify incoming data before it's stored in a Log Analytics workspace. This article explains how to write a KQL query that transforms text log data and add the transformation to a data collection rule. |
| 13 | + |
| 14 | +The procedure described here assumes you've already ingested some data from a text file, as described in [Collect text logs with Azure Monitor Agent](../agents/data-collection-text-log.md). In this tutorial, you'll: |
| 15 | + |
| 16 | +1. Write a KQL query to transform ingested data. |
| 17 | +1. Modify the target table schema. |
| 18 | +1. Add the transformation to your data collection rule. |
| 19 | +1. Verify that the transformation works correctly. |
| 20 | + |
| 21 | +## Prerequisites |
| 22 | + |
| 23 | +To complete this procedure, you need: |
| 24 | + |
| 25 | +- Log Analytics workspace where you have at least [contributor rights](../logs/manage-access.md#azure-rbac). |
| 26 | +- [Data collection rule](../essentials/data-collection-rule-overview.md), [data collection endpoint](../essentials/data-collection-endpoint-overview.md#create-a-data-collection-endpoint), and [custom table](../logs/create-custom-table.md#create-a-custom-table), as described in [Collect text logs with Azure Monitor Agent](../agents/data-collection-text-log.md). |
| 27 | +- A VM, Virtual Machine Scale Set, or Arc-enabled on-premises server that writes logs to a text file. |
| 28 | + Text file requirements: |
| 29 | + - Store on the local drive of the machine on which Azure Monitor Agent is running. |
| 30 | + - Delineate with an end of line. |
| 31 | + - Use ASCII or UTF-8 encoding. Other formats such as UTF-16 aren't supported. |
| 32 | + - Don't allow circular logging, log rotation where the file is overwritten with new entries, or renaming where a file is moved and a new file with the same name is opened. |
| 33 | + |
| 34 | + |
| 35 | +## Write a KQL query to transform ingested data |
| 36 | + |
| 37 | +1. View the data in the target custom table in Log Analytics: |
| 38 | + 1. In the Azure portal, select **Log Analytics workspaces** > your Log Analytics workspace > **Logs**. |
| 39 | + 1. Run a basic query the custom logs table to view table data. |
| 40 | +1. Use the query window to write and test a query that transforms the raw data in your table. |
| 41 | + |
| 42 | + For information about the KQL operators that transformations support, see [Structure of transformation in Azure Monitor](../essentials/data-collection-transformations-structure.md#kql-limitations). |
| 43 | + |
| 44 | + **Example** |
| 45 | + |
| 46 | + The sample uses [basic KQL operators](/azure/data-explorer/kql-quick-reference) to parse the data in the `RawData` column into three new columns, called `Time Ingested`, `RecordNumber`, and `RandomContent`: |
| 47 | + |
| 48 | + - The `extend` operator adds new columns. |
| 49 | + - The `project` operator formats the output to match the columns of the target table schema: |
| 50 | + |
| 51 | + ```kusto |
| 52 | + MyTable_CL |
| 53 | + | extend d=todynamic(RawData) |
| 54 | + | project TimeGenerated,TimeIngested=tostring(d.Time), |
| 55 | + RecordNumber=tostring(d.RecordNumber), |
| 56 | + RandomContent=tostring(d.RandomContent), |
| 57 | + RawData |
| 58 | + ``` |
| 59 | + > [!NOTE] |
| 60 | + > Information the user should notice even if skimmingQuerying table data in this way doesn't actually modify the data in the table. Azure Monitor applies the transformation in the [data ingestion pipeline](../essentials/data-collection-transformations.md#how-transformations-work) after you [add your transformation query to the data collection rule](#apply-the-transformation-to-your-data-collection-rule). |
| 61 | +
|
| 62 | +1. Format the query into a single line and replace the table name in the first line of the query with the word `source`. |
| 63 | + |
| 64 | + For example: |
| 65 | +
|
| 66 | + ```kusto |
| 67 | + source | extend d=todynamic(RawData) | project TimeGenerated,TimeIngested=tostring(d.Time),RecordNumber=tostring(d.RecordNumber), RandomContent=tostring(d.RandomContent), RawData |
| 68 | + ``` |
| 69 | +
|
| 70 | +1. Copy the formatted query so you can paste it into the data collection rule configuration. |
| 71 | +
|
| 72 | +## Modify the custom table to include the new columns |
| 73 | +
|
| 74 | +[Add or delete columns in your custom table](../logs/create-custom-table.md#add-or-delete-a-custom-column), based on your transformation query. |
| 75 | +
|
| 76 | +The example transformation query above adds three new columns of type `string`: |
| 77 | +- `TimeIngested` |
| 78 | +- `RecordNumber` |
| 79 | +- `RandomContent` |
| 80 | +
|
| 81 | +To support this transformation, add these three new columns to your custom table. |
| 82 | +
|
| 83 | +:::image type="content" source="media/azure-monitor-agent-transformation/add-custom-columns-azure-monitor-logs.png" alt-text="Screenshot of the Schema editor pane with the TimeIngested, RecordNumber, and RandomContent columns being defined." lightbox="media/azure-monitor-agent-transformation/add-custom-columns-azure-monitor-logs.png"::: |
| 84 | +
|
| 85 | +## Apply the transformation to your data collection rule |
| 86 | +
|
| 87 | +1. On the **Monitor** menu, select **Data Collection Rules** > your data collection rule. |
| 88 | +1. Select **Data sources** > your data source. |
| 89 | +1. Paste the formatted transformation query in the **Transform** field on the **Data source** tab of the **Add data source** screen. |
| 90 | +1. Select **Save**. |
| 91 | +
|
| 92 | + :::image type="content" source="media/azure-monitor-agent-transformation/add-transformation-to-data-collection-rule.png" alt-text="Screenshot of the Add data sources pane with the Transform field highlighted." lightbox="media/azure-monitor-agent-transformation/add-transformation-to-data-collection-rule.png"::: |
| 93 | +
|
| 94 | +## Check that the transformation works |
| 95 | +
|
| 96 | +View the data in the target custom table and check that data is being ingested correctly into the modified table: |
| 97 | +1. In the Azure portal, select **Log Analytics workspaces** > your Log Analytics workspace > **Logs**. |
| 98 | +1. Run a basic query the custom logs table to view table data. |
| 99 | +
|
| 100 | +
|
| 101 | +## Next steps |
| 102 | +
|
| 103 | +Learn more about: |
| 104 | +- [Data collection transformations](../essentials/data-collection-rule-structure.md). |
| 105 | +- [Data collection rules](../essentials/data-collection-rule-overview.md). |
| 106 | +- [Data collection endpoints](../essentials/data-collection-endpoint-overview.md). |
| 107 | +
|
| 108 | + |
0 commit comments