You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: data-explorer/includes/kusto-emulator-limitations.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
---
2
2
ms.service: azure
3
3
ms.topic: include
4
-
ms.date: 08/14/2022
4
+
ms.date: 02/05/2025
5
5
---
6
6
7
7
## Limitations
@@ -21,7 +21,7 @@ In general, this offering isn't suited for production workloads.
21
21
- No support for ingestion endpoints, including Kusto.Ingest SDKs
22
22
- No streaming ingestion
23
23
- Although ingested data can be stored externally to the container, we don't recommend persisting data for long periods of time for the following reasons:
24
-
- There's no guarantee the [extent](/kusto/management/extents-overview?view=azure-data-explorer&preserve-view=true) format will be compatible between versions of the Kusto emulator
24
+
- There's no guarantee the [extent](/kusto/management/extents-overview?view=azure-data-explorer&preserve-view=true) format is compatible between versions of the Kusto emulator
25
25
- Extents aren't [merged](/kusto/management/merge-policy?view=azure-data-explorer&preserve-view=true) and therefore can become fragmented as data get ingested
26
-
- Retention policies can be set but won't be honored
26
+
- Retention and partitioning policies can be set but aren't honored
27
27
- The [Python plugin](/kusto/query/python-plugin?view=azure-data-explorer&preserve-view=true?view=azure-data-explorer&preserve-view=true) isn't supported
Copy file name to clipboardExpand all lines: data-explorer/kusto-emulator-install.md
+55-33Lines changed: 55 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,28 +1,30 @@
1
1
---
2
2
title: Install the Azure Data Explorer Kusto emulator
3
-
description: In this article, you'll learn how to install the Azure Data Explorer Kusto emulator and run your first query.
3
+
description: In this article, you learn how to install the Azure Data Explorer Kusto emulator and run your first query.
4
4
ms.reviewer: vplauzon
5
5
ms.topic: how-to
6
-
ms.date: 12/05/2023
6
+
ms.date: 02/09/2025
7
7
---
8
8
9
9
# Install the Azure Data Explorer Kusto emulator
10
10
11
11
You can install the Azure Data Explorer Kusto emulator in the following ways:
12
12
13
-
- On your own device: Consider using this option if you need to provision a local development environment
14
-
- On a CI/CD agent virtual machine (VM): Use this option if you require a CI/CD pipeline for running automated tests
13
+
-**On your own device**: Consider using this option if you need to provision a local development environment
14
+
-**On a CI/CD agent virtual machine (VM)**: Use this option if you require a CI/CD pipeline for running automated tests
15
15
16
-
The emulator is available as a *Windows* or *Linux* Docker container image.
16
+
The emulator is available as a *Linux* and *Windows* Docker container image.
17
17
18
-
In this article, you'll learn how to:
18
+
In this article, you learn how to:
19
19
20
20
-[Install the Kusto emulator](#install-the-kusto-emulator)
21
21
-[Connect to the emulator](#connect-to-the-emulator)
22
22
-[Create a database](#create-a-database)
23
23
-[Ingest data](#ingest-data)
24
24
-[Query data](#query-data)
25
25
26
+
This article focuses on how to install the Linux Docker container on a Windows client.
27
+
26
28
## Prerequisites
27
29
28
30
- The host operating system must be one of:
@@ -35,25 +37,35 @@ In this article, you'll learn how to:
35
37
> Linux distros only support Linux container images.
36
38
37
39
- 2 gigabytes (GB) of RAM minimum; we recommend using 4 GB or more
38
-
- Docker Client for [Windows](https://docs.docker.com/desktop/windows/install/) or [Linux](https://docs.docker.com/desktop/install/linux-install/)
40
+
- Docker Client for [Linux](https://docs.docker.com/desktop/install/linux-install/) or [Windows](https://docs.docker.com/desktop/windows/install/)
39
41
40
42
## Install the Kusto emulator
41
43
42
44
The following steps are for using PowerShell to start the emulator using the [Kusto emulator container image](https://aka.ms/adx.emulator.image). For other options, see [Run emulator options](#run-emulator-options).
43
45
44
-
1. For Windows only, switch Docker to run with Windows containers. You may need to enable the feature in the Docker settings.
46
+
1.**For Windows container only**: Switch Docker to run with Windows containers. You might need to enable the feature in the Docker settings.
45
47
46
48
:::image type="content" source="media/kusto-emulator/kusto-emulator-docker-windows-container.png" alt-text="Screenshot of the Docker settings, showing the Switch to Windows containers option.":::
47
49
48
50
1. Run the following command to start the emulator.
49
51
50
52
> [!IMPORTANT]
51
-
> The Kusto emulator container image is a free offering under the [Microsoft Software License Terms](https://aka.ms/adx.emulator.license). Since the emulator runs in a container, you must accept the license terms by passing the `ACCEPT_EULA` environment variable to the container with its value set to `Y` indicating.
53
+
> The Kusto emulator container image is a free offering under the [Microsoft Software License Terms](https://aka.ms/adx.emulator.license). Since the emulator runs in a container, you must accept the license terms by passing the `ACCEPT_EULA` environment variable to the container with its value set to `Y`.
52
54
53
55
> [!NOTE]
54
56
>
55
-
> - The first time this command is run, Docker pulls the container image which is several GBs in size and may take several minutes to download. Once downloaded, the image is cached and available for subsequent runs without having to download it again.
56
-
> - (For Windows container only) The container must be run in process-isolation mode. This is the default on some versions of Docker. For other versions, you can start the container in Hyper-V isolation mode by adding `--isolation=hyperv` to the run command.
57
+
> - The first time this command is run, Docker pulls the container image which is several GBs in size and might take several minutes to download. Once downloaded, the image is cached and available for subsequent runs without having to download it again.
58
+
> -**For Windows container only**: The container must run in process-isolation mode. This is the default on some versions of Docker. For other versions, you can start the container in Hyper-V isolation mode by adding `--isolation=hyperv` to the run command.
59
+
60
+
# [Linux container](#tab/linux-container)
61
+
62
+
To start the Linux container, make sure you use the `latest` or `stable` tag:
1. Run the following command to verify that the container is running.
77
85
@@ -114,24 +122,35 @@ The following steps are for using PowerShell to start the emulator using the [Ku
114
122
RawContentLength : 988
115
123
```
116
124
117
-
### Run emulator options
125
+
## Run emulator options
118
126
119
127
You can use any of the following options when running the emulator:
120
128
121
129
- Mount a local folder to the container: Use this option to mount a folder in the host environment into the container. Mounting a host folder enables your queries to interact with local files, which is useful for [creating a database persistent between container runs](#create-a-database) and [ingesting data](#ingest-data).
122
130
123
-
For example, to mount the folder "D:\host\local" on the host to the folder "c:\kustodatadata" in the container, use the next command on Windows Server:
131
+
# [Linux container](#tab/linux-container)
132
+
133
+
For example, to mount the folder "D:\host\local" on the host to the folder "/kustodatadata" in the container, use the following command on Windows Server:
For example, to mount the folder "D:\host\local" on the host to the folder "c:\kustodata" in the container, use the following command on Windows Server:
- Run on a different port: The Kusto emulator exposes access to the Kusto Query Engine on port 8080; hence in other examples you mapped the host port 8080 to the emulator port 8080. You can use this option to map a different host to the engine.
130
149
131
150
For example, to map port 9000 on the host to the engine, use the following command on Windows Server:
In the following sections, you'll use Kusto.Explorer to create a database, ingest data, and query it. To learn more, see [Using Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true).
164
+
In the following sections, you use Kusto.Explorer to create a database, ingest data, and query it. To learn more, see [Using Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true).
146
165
147
166
> [!IMPORTANT]
148
167
> The Kusto Emulator doesn't support HTTPS or Microsoft Entra authentication.
149
168
>
150
169
> You must first allow unsafe connections in **Tools** > **Options** > **Connections** > **Allow unsafe connections**.
151
-
>
170
+
>
152
171
> :::image type="content" source="media/kusto-emulator/unsafe-connections.png" alt-text="Screenshot of Kusto Explorer connection options.":::
153
-
>
172
+
>
154
173
> The following image highlights the affected fields in the **Add connection** properties.
155
174
>
156
175
> - The **Cluster connection** must begin with `http://` and not `https://`.
157
-
> - In **Security** > **Advanced: Connection String**, you'll need to remove the `AAD Federated Security=True` portion of the connection string to disable Microsoft Entra authentication.
176
+
> - In **Security** > **Advanced: Connection String**, you need to remove the `AAD Federated Security=True` portion of the connection string to disable Microsoft Entra authentication.
158
177
>
159
178
> :::image type="content" source="media/kusto-emulator/kusto-emulator-connection.png" alt-text="Screenshot of Kusto Explorer single connection setup.":::
160
179
161
180
## Create a database
162
181
163
182
To store data and run queries, create a database or attach the emulator to an existing database.
164
183
165
-
A database can be persisted in a container folder or on a [mounted folder](#run-emulator-options). The former's lifetime is bound to the container, so restarting the container loses any changes. Also, the container virtual storage is less efficient than native one. Mounted folder enables you to keep the data between container runs.
184
+
A database can be persisted in a container folder or on a [mounted folder](#run-emulator-options). The former's lifetime is bound to the container, so restarting the container loses any changes. Also, the container virtual storage is less efficient than native one. Mounted folder enables you to keep the data between container runs.
166
185
167
186
In this example, we keep the data on the container.
168
187
169
-
In the [Kusto.Explorer Query mode](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true#query-mode), run the following command to create a persistent database if using a Windows container image:
188
+
In the [Kusto.Explorer Query mode](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true#query-mode), run the following command to create a persistent database:
189
+
190
+
# [Linux container](#tab/linux-container)
170
191
171
192
```kusto
172
193
.create database <YourDatabaseName> persist (
173
-
@"c:\kustodata\dbs\<YourDatabaseName>\md",
174
-
@"c:\kustodata\dbs\<YourDatabaseName>\data"
194
+
@"/kustodata/dbs/<YourDatabaseName>/md",
195
+
@"/kustodata/dbs/<YourDatabaseName>/data"
175
196
)
176
197
```
177
198
178
-
The equivalent command for a Linux container image is:
199
+
# [Windows container](#tab/windowscontainer)
179
200
180
201
```kusto
181
202
.create database <YourDatabaseName> persist (
182
-
@"/kustodata/dbs/<YourDatabaseName>/md",
183
-
@"/kustodata/dbs/<YourDatabaseName>/data"
203
+
@"c:\kustodata\dbs\<YourDatabaseName>\md",
204
+
@"c:\kustodata\dbs\<YourDatabaseName>\data"
184
205
)
185
206
```
207
+
---
186
208
187
209
This command requires that the folders don't already exist, to prevent over-writing existing information. To attach to an existing database, use the following command instead, specifying the path that ends with `md`:
188
210
189
211
```kusto
190
212
.attach database <YourDatabaseName> from @"<PathToMdFolder>"
191
213
```
192
214
193
-
It's also possible to detach the database from the emulator, which will keep all the database metadata and data intact (so you could reattach to it in the future):
215
+
It's also possible to detach the database from the emulator, which will keep all the database metadata and data intact so you can reattach to it in the future:
194
216
195
217
```kusto
196
218
.detach database <YourDatabaseName>
@@ -200,7 +222,7 @@ It's also possible to detach the database from the emulator, which will keep all
200
222
201
223
To ingest data, you must first create an external table linked to a file and then ingest the data into a table in the database.
202
224
203
-
Use the steps in the following example to create an external table and ingest data into it. For the example, in the local folder *c:\kustodata*, create a file called `sample.csv` with the following data:
225
+
Use the steps in the following example to create an external table and ingest data into it. As part of the example, create a file called `sample.csv` in the local folder *d:\host\local*, with the following data:
204
226
205
227
```text
206
228
Alice, 1
@@ -217,7 +239,7 @@ Carl, 3
217
239
1. Run the following command to [ingest the file into the table](/kusto/management/data-ingestion/ingest-from-storage?view=azure-data-explorer&preserve-view=true):
218
240
219
241
```kusto
220
-
.ingest into table MyIngestedSample(@"c:\kustodata\sample.csv")
242
+
.ingest into table MyIngestedSample(@"/kustodata/sample.csv")
Copy file name to clipboardExpand all lines: data-explorer/kusto-emulator-overview.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,9 @@
1
1
---
2
2
title: Azure Data Explorer Kusto emulator overview
3
-
description: In this article, you'll learn about the Azure Data Explorer Kusto emulator
3
+
description: In this article, you learn about the Azure Data Explorer Kusto emulator.
4
4
ms.reviewer: vplauzon
5
5
ms.topic: conceptual
6
-
ms.date: 05/08/2023
6
+
ms.date: 02/06/2025
7
7
---
8
8
# Azure Data Explorer Kusto emulator
9
9
@@ -13,11 +13,11 @@ The Kusto emulator is a local environment that encapsulates the query engine. Yo
13
13
>
14
14
> - The emulator is provided *as-is*, without any support or warranties.
15
15
> - The emulator isn't intended for use in production environments.
16
-
> - The [license terms](https://aka.ms/adx.emulator.license) explicitly prohibit benchmark tests using the emulator as it is not intended or optimized for that purpose. The emulator has a very different performance profile compared to the Azure Data Explorer service.
16
+
> - The [license terms](https://aka.ms/adx.emulator.license) explicitly prohibit benchmark tests using the emulator as it isn't intended or optimized for that purpose. The emulator has a different performance profile compared to the Azure Data Explorer service.
17
17
18
18
## Architecture
19
19
20
-
The emulator is available as a *Windows* or *Linux* Docker container image. It exposes a query endpoint over HTTP that can be used with any client including [Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true), [Kusto.CLI](/kusto/tools/kusto-cli?view=azure-data-explorer&preserve-view=true), or [Kusto.Data SDKs](/kusto/api/netfx/about-kusto-data?view=azure-data-explorer&preserve-view=true).
20
+
The emulator is available as a *Linux* and *Windows* Docker container image. It exposes a query endpoint over HTTP that can be used with any client including [Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true), [Kusto.CLI](/kusto/tools/kusto-cli?view=azure-data-explorer&preserve-view=true), or [Kusto.Data SDKs](/kusto/api/netfx/about-kusto-data?view=azure-data-explorer&preserve-view=true).
21
21
22
22

23
23
@@ -34,7 +34,7 @@ The main scenarios for the emulator are:
34
34
35
35
## Kusto emulator vs. free cluster
36
36
37
-
There are some overlaps between the Kusto emulator and the [free cluster](start-for-free.md) offerings. The following table contrasts the features of the offerings.
37
+
There are some overlaps between the Kusto emulator and the [free cluster](start-for-free.md) offerings. The following table contrasts the features of the offerings.
38
38
39
39
| Item | Kusto emulator | Free cluster |
40
40
|--|--|--|
@@ -49,10 +49,10 @@ There are some overlaps between the Kusto emulator and the [free cluster](start-
49
49
| Long term data management | No extent merge capability | Full support |
50
50
| Internet requirement | No connectivity required | Cloud service |
51
51
52
-
When choosing the best option for your use case, you should consider the following:
52
+
When choosing the best option for your use case, you should consider:
53
53
54
54
-**Local development**: The best fit for local development depends on the features you require. For instance, if your scenario requires the use of managed pipelines, the free cluster offering works best. On the other hand, if your scenario requires local development in a disconnected environment, the Kusto emulator would be a better fit.
55
-
-**Automated testing**: In general, the Kusto emulator is more suited to automated testing than the free cluster offering. It's faster to provision and doesn't require any Azure principal to set up.
55
+
-**Automated testing**: In general, the Kusto emulator is more suited to automated testing than the free cluster offering. It's faster to provision and doesn't require any Azure principal to set up.
> The examples in this article use publicly available tables in the [help cluster](https://dataexplorer.azure.com/clusters/help/), such as the `StormEvents` table in the *Samples* database.
0 commit comments