You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/time-series-insights/how-to-tsi-gen1-migration.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,25 +26,25 @@ ms.custom: tvilutis
26
26
27
27
Use the following steps to start ingesting new data into your Eventhouse:
28
28
29
-
1. Configure your [Azure Event Hub](/azure/event-hubs/event-hubs-about) with a new consumer group.
29
+
1. Configure your [event hub](/azure/event-hubs/event-hubs-about) with a new consumer group.
30
30
31
-
2. Consume data from the data source and ingest it into your Eventhouse. Refer to the documentation on how to [ingest data from your EventHub](/fabric/real-time-intelligence/get-data-event-hub).
31
+
2. Consume data from the data source and ingest it into your Eventhouse. Refer to the documentation on how to [ingest data from your event hub](/fabric/real-time-intelligence/get-data-event-hub).
32
32
33
33
### Migrate historical data from Time Series Insights
34
34
35
-
If you need to export telemetry data from your Time Series Insights environment, you can use the Time Series Insights Query API to download the events in batches and serialize them in the required format. Depending on where you stored the exported data, you can ingest the data from [Azure Storage](/real-time-intelligence/get-data-azure-storage), [local files](/fabric/real-time-intelligence/get-data-local-file), or [OneLake](/fabric/real-time-intelligence/get-data-onelake).
35
+
If you need to export data from your Time Series Insights environment, you can use the Time Series Insights Query API to download the events in batches and serialize them in the required format. Depending on where you stored the exported data, you can ingest the data from [Azure Storage](/real-time-intelligence/get-data-azure-storage), [local files](/fabric/real-time-intelligence/get-data-local-file), or [OneLake](/fabric/real-time-intelligence/get-data-onelake).
36
36
37
37
### Migrate reference data
38
38
39
39
Use the following steps to migrate reference data:
40
40
41
-
1. Use Time Series Insights Explorer or the Reference Data API to download the reference data set. This allows you to retrieve the necessary data for migration.
41
+
1. Use Time Series Insights Explorer or the Reference Data API to download the reference data set.
42
42
43
43
2. Once you have the reference data set, [upload it to your Eventhouse](/fabric/real-time-intelligence/get-data-local-file) as another table. By uploading the reference data set, you can access and utilize it within your Eventhouse environment.
44
44
45
-
## Translate Time Series Insights Queries to KQL
45
+
## Translate Time Series Insights Queries to Kusto Query Language
46
46
47
-
For queries, the recommendation is to use KQL in Eventhouse.
47
+
For queries, the recommendation is to use Kusto Query Language in Eventhouse.
| Ingesting JSON from Hub with flattening and escaping |[Get data from Azure Event Hubs](/fabric/real-time-intelligence/get-data-event-hub)|
25
25
| Open Cold store |[Eventhouse OneLake Availability](/fabric/real-time-intelligence/event-house-onelake-availability)|
26
-
|PBI Connector | Use [Eventhouse Power BI Connector](/fabric/real-time-intelligence/create-powerbi-report). Rewrite TSQ to KQL manually. |
26
+
|Power BI Connector | Use [Eventhouse Power BI Connector](/fabric/real-time-intelligence/create-powerbi-report). Rewrite TSQ to KQL manually. |
27
27
| Spark Connector | Migrate data to Eventhouse. [Use a notebook with Apache Spark to query an Eventhouse](/fabric/real-time-intelligence/spark-connector) or [Explore the data in your lakehouse with a notebook](/fabric/data-engineering/lakehouse-notebook-explore)|
28
28
| Bulk Upload |[Get data from Azure storage](/fabric/real-time-intelligence/get-data-azure-storage)|
29
-
| Time Series Model | Can be exported as JSON file. Can be [imported](/fabric/real-time-intelligence/get-data-local-file) to Eventhouse. [Kusto Graph Semantics](/azure/data-explorer/graph-overview?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric) allow model, traverse and analyze Time Series Model hierarchy as a graph |
30
-
|TSI Explorer |[Real-Time Dashboard](/fabric/real-time-intelligence/dashboard-real-time-create), [Power BI report](/fabric/real-time-intelligence/create-powerbi-report) or write a custom dashboard using [KustoTrender](https://github.com/Azure/azure-kusto-trender)|
29
+
| Time Series Model | Can be exported as JSON file. Can be [imported](/fabric/real-time-intelligence/get-data-local-file) to Eventhouse. [Kusto Graph Semantics](/azure/data-explorer/graph-overview?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric) allow model, traverse, and analyze Time Series Model hierarchy as a graph |
30
+
|Time Series Explorer |[Real-Time Dashboard](/fabric/real-time-intelligence/dashboard-real-time-create), [Power BI report](/fabric/real-time-intelligence/create-powerbi-report) or write a custom dashboard using [KustoTrender](https://github.com/Azure/azure-kusto-trender)|
31
31
| Query language | Rewrite queries in KQL. |
32
32
33
33
## Migrating Telemetry
34
34
35
-
Use `PT=Time` folder in the storage account to retrieve the copy of all telemetry in the environment. For more information, please see [Data Storage](./concepts-storage.md#cold-store).
35
+
To retrieve the copy of all data in the environment, use `PT=Time` folder in the storage account. For more information, please see [Data Storage](./concepts-storage.md#cold-store).
36
36
37
37
### Migration Step 1 – Get Statistics about Telemetry Data
38
38
@@ -41,48 +41,48 @@ Data
41
41
- Record Environment ID from first part of Data Access FQDN (for example, d390b0b0-1445-4c0c-8365-68d6382c1c2a From .env.crystal-dev.windows-int.net)
- Record size and the number of blobs of `PT=Time` folder. For customers in private preview of Bulk Import, also record `PT=Import` size and number of blobs.
44
+
- Record size and the number of blobs of `PT=Time` folder.
45
45
46
46
47
-
### Migration Step 2 – Migrate Telemetry To Eventhouse
47
+
### Migration Step 2 – Migrate Data To Eventhouse
48
48
49
49
#### Create an Eventhouse
50
50
51
-
Follow the steps in [creating an Eventhouse](/fabric/real-time-intelligence/create-eventhouse) to lset up an Eventhouse for your migration process.
51
+
To set up an Eventhouse for your migration process, follow the steps in [creating an Eventhouse](/fabric/real-time-intelligence/create-eventhouse).
52
52
53
53
#### Data Ingestion
54
54
55
-
Follow the steps in [getting data from Azure Storage](/fabric/real-time-intelligence/get-data-azure-storage) to retrieve data for the storage account corresponding to your Time Series Insights instance.
55
+
To retrieve data for the storage account corresponding to your Time Series Insights instance, follow the steps in [getting data from Azure Storage](/fabric/real-time-intelligence/get-data-azure-storage).
56
56
57
57
Make sure that you:
58
58
59
59
1. Select the appropriate container and provide its URI, along with the necessary [SAS token](/azure/data-explorer/kusto/api/connection-strings/storage-connection-strings#shared-access-sas-token) or [account key](/azure/data-explorer/kusto/api/connection-strings/storage-connection-strings#storage-account-access-key).
60
60
61
61
1. Configure file filters folder path as `V=1/PT=Time` to filter the relevant blobs.
62
-
1. Verify the inferred schema and remove any infrequently queried columns, while retaining at least the timestamp, TSID column(s), and values. To ensure that all your data is copied to Eventhouse add an additional column and use the [DropMappedFields](/azure/data-explorer/kusto/management/mappings#dropmappedfields-transformation) mapping transformation.
62
+
1. Verify the inferred schema and remove any infrequently queried columns, while retaining at least the timestamp, TSID columns, and values. To ensure that all your data is copied to Eventhouse add another column and use the [DropMappedFields](/azure/data-explorer/kusto/management/mappings#dropmappedfields-transformation) mapping transformation.
63
63
1. Complete the ingestion process.
64
64
65
65
#### Querying the data
66
66
67
-
Now that you have successfully ingested the data, you can begin exploring it using a [KQL query set](/fabric/real-time-intelligence/create-query-set). If you need to access the data from your custom client application, Eventhouse provides SDKs for major programming languages such as C# ([link](/azure/data-explorer/kusto/api/netfx/about-the-sdk?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric)), Java ([link](/azure/data-explorer/kusto/api/java/kusto-java-client-library?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric)), and Node.js ([link](/azure/data-explorer/kusto/api/node/kusto-node-client-library?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric)).
67
+
Now that you successfully ingested the data, you can begin exploring it using a [KQL queryset](/fabric/real-time-intelligence/create-query-set). If you need to access the data from your custom client application, Eventhouse provides SDKs for major programming languages such as C# ([link](/azure/data-explorer/kusto/api/netfx/about-the-sdk?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric)), Java ([link](/azure/data-explorer/kusto/api/java/kusto-java-client-library?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric)), and Node.js ([link](/azure/data-explorer/kusto/api/node/kusto-node-client-library?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric)).
68
68
69
69
## Migrating Time Series Model to Azure Data Explorer
70
70
71
71
The model can be download in JSON format from TSI Environment using TSI Explorer UX or TSM Batch API.
72
72
Then the model can be imported to Eventhouse.
73
73
74
74
1. Download TSM from TSI UX.
75
-
1. Delete first three lines using VSCode or another editor.
75
+
1. Delete first three lines using Visual Studio Code or another editor.
76
76
77
77
:::image type="content" source="media/gen2-migration/adx-tsm-1.png" alt-text="Screenshot of TSM migration to the Azure Data Explorer - Delete first 3 lines" lightbox="media/gen2-migration/adx-tsm-1.png":::
78
78
79
-
1. Using VSCode or another editor, search and replace as regex `\},\n \{` with `}{`
79
+
1. Using Visual Studio Code or another editor, search and replace as regex `\},\n \{` with `}{`
80
80
81
81
:::image type="content" source="media/gen2-migration/adx-tsm-2.png" alt-text="Screenshot of TSM migration to the Azure Data Explorer - search and replace" lightbox="media/gen2-migration/adx-tsm-2.png":::
82
82
83
83
1. Ingest as JSON into ADX as a separate table using [Get data from a single file](/fabric/real-time-intelligence/get-data-local-file).
84
84
85
-
Once you have migrated your time series data to Eventhouse in Fabric Real-Time Intelligence, you can leverage the power of [Kusto Graph Semantics](/azure/data-explorer/graph-overview?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric) to contextualize and analyze your data. Kusto Graph Semantics allows you to model, traverse, and analyze the hierarchy of your Time Series Model as a graph. By using Kusto Graph Semantics, you can gain insights into the relationships between different entities in your time series data, such as assets, sites, and data points. This can help you understand the dependencies and interactions between various components of your system.
85
+
Once you migrated your time series data to Eventhouse in Fabric Real-Time Intelligence, you can use the power of [Kusto Graph Semantics](/azure/data-explorer/graph-overview?context=%2Ffabric%2Fcontext%2Fcontext-rti&pivots=fabric) to contextualize and analyze your data. Kusto Graph Semantics allows you to model, traverse, and analyze the hierarchy of your Time Series Model as a graph. By using Kusto Graph Semantics, you can gain insights into the relationships between different entities in your time series data, such as assets, sites, and data points. These insights help you to understand the dependencies and interactions between various components of your system.
86
86
87
87
## Translate Time Series Queries (TSQ) to KQL
88
88
@@ -268,22 +268,22 @@ events
268
268
269
269
## Power BI
270
270
271
-
There is no automated process for migrating Power BI reports that were based on Time Series Insights. All queries relying on data stored in Time Series Insights must be migrated to [Eventhouse](/fabric/real-time-intelligence/create-powerbi-report).
271
+
There's no automated process for migrating Power BI reports that were based on Time Series Insights. All queries relying on data stored in Time Series Insights must be migrated to [Eventhouse](/fabric/real-time-intelligence/create-powerbi-report).
272
272
273
273
To create efficient time series reports in Power BI, we recommend referring to the following informative blog articles:
274
274
275
-
-[Using Eventhouse timeseries capabilities in Power BI](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/using-azure-data-explorer-timeseries-capabilities-in-power-bi/ba-p/2727977)
275
+
-[Eventhouse time series capabilities in Power BI](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/using-azure-data-explorer-timeseries-capabilities-in-power-bi/ba-p/2727977)
276
276
-[How to use M dynamic parameters without most limitations](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/how-to-use-m-dynamic-parameters-without-most-limitations/ba-p/4117352)
277
-
-[Timespan/duration values in KQL, Power Query and Power BI](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/timespan-duration-values-in-kql-power-query-and-power-bi/ba-p/4086091)
278
-
-[Using KQL query settings in Power BI](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/using-kql-query-settings-in-power-bi/ba-p/4013580)
277
+
-[Timespan/duration values in KQL, Power Query, and Power BI](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/timespan-duration-values-in-kql-power-query-and-power-bi/ba-p/4086091)
278
+
-[KQL query settings in Power BI](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/using-kql-query-settings-in-power-bi/ba-p/4013580)
279
279
-[Filtering and visualizing Kusto data in local time](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/filtering-and-visualizing-kusto-data-in-local-time/ba-p/3948778)
280
280
-[Near real-time reports in PBI + Kusto](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/near-real-time-reports-in-pbi-kusto/ba-p/3884322)
281
281
-[Power BI modeling with ADX - cheat sheet](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/power-bi-modeling-with-adx-cheat-sheet/ba-p/3768392)
282
282
283
-
Please refer to these resources for guidance on creating effective time series reports in Power BI.
283
+
Refer to these resources for guidance on creating effective time series reports in Power BI.
284
284
285
285
## Real-Time Dashboard
286
286
287
287
A Real-Time Dashboard in Fabric is a collection of tiles, optionally organized in pages, where each tile has an underlying query and a visual representation. You can natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition to ease of data exploration, this fully integrated dashboard experience provides improved query and visualization performance.
288
288
289
-
Start by creating a new dashboard in Fabric Real-Time Intelligence. This powerful feature allows you to explore data, customize visuals, apply conditional formatting, and utilize parameters. Furthermore, you can create alerts directly from your Real-Time Dashboards, enhancing your monitoring capabilities. For detailed instructions on how to create a dashboard, please refer to the [official documentation](/fabric/real-time-intelligence/dashboard-real-time-create).
289
+
Start by creating a new dashboard in Fabric Real-Time Intelligence. This powerful feature allows you to explore data, customize visuals, apply conditional formatting, and utilize parameters. Furthermore, you can create alerts directly from your Real-Time Dashboards, enhancing your monitoring capabilities. For detailed instructions on how to create a dashboard, refer to the [official documentation](/fabric/real-time-intelligence/dashboard-real-time-create).
0 commit comments