Skip to content

Repo sync for protected branch #2624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
learn-build-service-prod merged 4 commits into from
May 19, 2025
Merged

Repo sync for protected branch #2624

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1f721f4
Adding external artifacts to fabric and other misc. fixes
adieldar May 19, 2025
432fea0
Merge pull request #6848 from adieldar/adieldar/python_update
v-dirichards May 19, 2025
ed1fde9
Merge pull request #6851 from MicrosoftDocs/main
Taojunshen May 19, 2025
952f8aa
Merging changes synced from https://github.com/MicrosoftDocs/dataexpl…
May 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 14 additions & 39 deletions data-explorer/kusto/includes/python-plugin-adx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
ms.topic: include
ms.date: 09/17/2024
ms.date: 05/19/2025
---

The Python plugin runs a user-defined function (UDF) using a Python script. The Python script gets tabular data as its input, and produces tabular output. The plugin's runtime is hosted in [sandboxes](../concepts/sandboxes.md), running on the cluster's nodes.

## Syntax

*T* `|` `evaluate` [`hint.distribution` `=` (`single` | `per_node`)] [`hint.remote` `=` (`auto` | `local`)] `python(`*output_schema*`,` *script* [`,` *script_parameters*] [`,` *external_artifacts*][`,` *spill_to_disk*]`)`
*T* `|` `evaluate` [`hint.distribution` `=` (`single` | `per_node`)] [`hint.remote` `=` (`auto` | `local`)] `python(`*output_schema*`,` *script* [`,` *script_parameters*] [`,` *external_artifacts*] [`,` *spill_to_disk*]`)`

[!INCLUDE [syntax-conventions-note](syntax-conventions-note.md)]

Expand Down Expand Up @@ -49,11 +49,11 @@ To see the list of packages for the different Python images, see [Python package
## Use ingestion from query and update policy

* Use the plugin in queries that are:
* Defined as part of an [update policy](../management/update-policy.md), whose source table is ingested to using *non-streaming* ingestion.
* Defined as part of an [update policy](../management/update-policy.md), whose source table is ingested by [queued ingestion](/azure/data-explorer/ingest-data-overview#continuous-data-ingestion).
* Run as part of a command that [ingests from a query](../management/data-ingestion/ingest-from-query.md), such as `.set-or-append`.
* You can't use the plugin in a query that is defined as part of an update policy, whose source table is ingested using [streaming ingestion](/azure/data-explorer/ingest-data-streaming).

## Examples
## Example

~~~kusto
range x from 1 to 360 step 1
Expand All @@ -74,31 +74,6 @@ result["fx"] = g * np.sin(df["x"]/n*2*np.pi*f)

:::image type="content" source="../query/media/plugin/sine-demo.png" alt-text="Screenshot of sine demo showing query result." border="false":::

~~~kusto
print "This is an example for using 'external_artifacts'"
| evaluate python(
typeof(File:string, Size:string), ```if 1:
import os
result = pd.DataFrame(columns=['File','Size'])
sizes = []
path = '.\\\\Temp'
files = os.listdir(path)
result['File']=files
for file in files:
sizes.append(os.path.getsize(path + '\\\\' + file))
result['Size'] = sizes
```,
external_artifacts =
dynamic({"this_is_my_first_file":"https://kustoscriptsamples.blob.core.windows.net/samples/R/sample_script.r",
"this_is_a_script":"https://kustoscriptsamples.blob.core.windows.net/samples/python/sample_script.py"})
)
~~~

| File | Size |
|--------|------|
| this_is_a_script | 120 |
| this_is_my_first_file | 105 |

## Performance tips

* Reduce the plugin's input dataset to the minimum amount required (columns/rows).
Expand All @@ -116,9 +91,9 @@ print "This is an example for using 'external_artifacts'"
` ``` `
` python code`
` ``` `
* Use the [`externaldata` operator](../query/externaldata-operator.md) to obtain the content of a script that you've stored in an external location, such as Azure Blob storage.
* Use the [externaldata operator](../query/externaldata-operator.md) to obtain the content of a script that you've stored in an external location, such as Azure Blob storage.

### Example
### Example reading the Python script external data

```kusto
let script =
Expand All @@ -145,7 +120,7 @@ The URLs referenced by the external artifacts property must be:
> [!NOTE]
> When authenticating external artifacts using Managed Identities, the `SandboxArtifacts` usage must be defined on the cluster level [managed identity policy](../management/managed-identity-policy.md).

The artifacts are made available for the script to consume from a local temporary directory, `.\Temp`. The names provided in the property bag are used as the local file names. See [Examples](#examples).
The artifacts are made available for the script to be read from a local temporary directory, `.\Temp`. The names provided in the property bag are used as the local file names. See [Example](#example-using-external-artifacts).

For information regarding referencing external packages, see [Install packages for the Python plugin](#install-packages-for-the-python-plugin).

Expand Down Expand Up @@ -188,24 +163,24 @@ download the package and its dependencies.
pip wheel [-w download-dir] package-name.
```

1. Create a ZIP file that contains the required package and its dependencies.
1. Create a zip file containing the required package and its dependencies.

* For private packages, zip the folder of the package and the folders of its dependencies.
* For public packages, zip the files that were downloaded in the previous step.

> [!NOTE]
>
> * Make sure to download the package that is compatible to the Python engine and the platform of the sandbox runtime (currently 3.6.5 on Windows)
> * Make sure to download the package that is compatible to the Python engine and the platform of the sandbox runtime (currently 3.10.8 or 3.11.7 on Windows)
> * Make sure to zip the `.whl` files themselves, and not their parent folder.
> * You can skip `.whl` files for packages that already exist with the same version in the base sandbox image.

1. Upload the zipped file to a blob in the artifacts location (from step 1).
1. Upload the zip file to a blob in the artifacts location (from step 1 of the prerequisites).

1. Call the `python` plugin.
* Specify the `external_artifacts` parameter with a property bag of name and reference to the ZIP file (the blob's URL, including a SAS token).
* In your inline python code, import `Zipackage` from `sandbox_utils` and call its `install()` method with the name of the ZIP file.
* Specify the `external_artifacts` parameter with a property bag of local name and blob URL of the zip file (including a SAS token).
* In your inline python code, import `Zipackage` from `sandbox_utils` and call its `install()` method with the local name of the ZIP file.

### Example
### Example using external artifacts

Install the [Faker](https://pypi.org/project/Faker/) package that generates fake data.

Expand All @@ -221,7 +196,7 @@ range ID from 1 to 3 step 1
for i in range(df.shape[0]):
result.loc[i, "Name"] = fake.name()
```,
external_artifacts=bag_pack('faker.zip', 'https://artifacts.blob.core.windows.net/Faker.zip?*** REPLACE WITH YOUR SAS TOKEN ***'))
external_artifacts=bag_pack('faker.zip', 'https://artifacts.blob.core.windows.net/Faker.zip;impersonate'))
~~~

| ID | Name |
Expand Down
91 changes: 84 additions & 7 deletions data-explorer/kusto/includes/python-plugin-fabric.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
---
ms.topic: include
ms.date: 08/11/2024
ms.date: 05/19/2025
---

The Python plugin runs a user-defined function (UDF) using a Python script. The Python script gets tabular data as its input, and produces tabular output.

## Syntax

*T* `|` `evaluate` [`hint.distribution` `=` (`single` | `per_node`)] [`hint.remote` `=` (`auto` | `local`)] `python(`*output_schema*`,` *script* [`,` *script_parameters*] [`,` *spill_to_disk*]`)`
*T* `|` `evaluate` [`hint.distribution` `=` (`single` | `per_node`)] [`hint.remote` `=` (`auto` | `local`)] `python(`*output_schema*`,` *script* [`,` *script_parameters*] [`,` *external_artifacts*] [`,` *spill_to_disk*]`)`


[!INCLUDE [syntax-conventions-note](syntax-conventions-note.md)]

Expand All @@ -18,8 +19,9 @@ The Python plugin runs a user-defined function (UDF) using a Python script. The
|*output_schema*| `string` | :heavy_check_mark:|A `type` literal that defines the output schema of the tabular data, returned by the Python code. The format is: `typeof(`*ColumnName*`:` *ColumnType*[, ...]`)`. For example, `typeof(col1:string, col2:long)`. To extend the input schema, use the following syntax: `typeof(*, col1:string, col2:long)`.|
|*script*| `string` | :heavy_check_mark:|The valid Python script to execute. To generate multi-line strings, see [Usage tips](#usage-tips).|
|*script_parameters*| `dynamic` ||A property bag of name value pairs to be passed to the Python script as the reserved `kargs` dictionary. For more information, see [Reserved Python variables](#reserved-python-variables).|
|*hint.distribution*| `string` ||A hint for the plugin's execution to be distributed across multiple cluster nodes. The default value is `single`. `single` means a single instance of the script will run over the entire query data. `per_node` means that if the query before the Python block is distributed, an instance of the script will run on each node, on the data that it contains.|
|*hint.distribution*| `string` ||A hint for the plugin's execution to be distributed across multiple sandboxes. The default value is `single`. `single` means a single instance of the script will run over the entire query data in a single sandbox. `per_node` means that if the query before the Python block is distributed to partitions, each partition will run in its own sandbox in parallel.|
|*hint.remote*| `string` ||This hint is only relevant for cross cluster queries. The default value is `auto`. `auto` means the server decides automatically in which cluster the Python code is executed. Setting the value to `local` forces executing the Python code on the local cluster. Use it in case the Python plugin is disabled on the remote cluster.|
|*external_artifacts*| `dynamic` ||A property bag of name and URL pairs for artifacts that are accessible from OneLake storage. See more in [Using external artifacts](#using-external-artifacts).|
|*spill_to_disk*| `bool` ||Specifies an alternative method for serializing the input table to the Python sandbox. For serializing big tables set it to `true` to speed up the serialization and significantly reduce the sandbox memory consumption. Default is `true`.|

## Reserved Python variables
Expand All @@ -46,11 +48,11 @@ To see the list of packages for the different Python images, see [Python package
## Use ingestion from query and update policy

* Use the plugin in queries that are:
* Defined as part of an [update policy](../management/update-policy.md), whose source table is ingested to using *non-streaming* ingestion.
* Defined as part of an [update policy](../management/update-policy.md), whose source table is ingested by [queued ingestion](/azure/data-explorer/ingest-data-overview#continuous-data-ingestion).
* Run as part of a command that [ingests from a query](../management/data-ingestion/ingest-from-query.md), such as `.set-or-append`.
* You can't use the plugin in a query that is defined as part of an update policy, whose source table is ingested using [streaming ingestion](/azure/data-explorer/ingest-data-streaming).

## Examples
## Example

~~~kusto
range x from 1 to 360 step 1
Expand Down Expand Up @@ -88,9 +90,9 @@ result["fx"] = g * np.sin(df["x"]/n*2*np.pi*f)
` ``` `
` python code`
` ``` `
* Use the [`externaldata` operator](../query/externaldata-operator.md) to obtain the content of a script that you've stored in an external location, such as Azure Blob storage.
* Use the [externaldata operator](../query/externaldata-operator.md) to obtain the content of a script that you've stored in an external location, such as Azure Blob storage.

### Example
### Example reading the Python script external data

```kusto
let script =
Expand All @@ -105,6 +107,81 @@ result["fx"] = g * np.sin(df["x"]/n*2*np.pi*f)
| render linechart
```

## Using External Artifacts

External artifacts from OneLake storage can be made available for the script and used at runtime.

The artifacts are made available for the script to be read from a local temporary directory, `.\Temp`. The names provided in the property bag are used as the local file names. See [Example](#example-using-external-artifacts).

For information regarding referencing external packages, see [Install packages for the Python plugin](#install-packages-for-the-python-plugin).

### Refreshing external artifact cache

External artifact files utilized in queries are cached on your cluster. If you make updates to your files in cloud storage and require immediate synchronization with your cluster, you can use the [.clear cluster cache external-artifacts command](../management/clear-external-artifacts-cache-command.md). This command clears the cached files and ensures that subsequent queries run with the latest version of the artifacts.

## Install packages for the Python plugin

Install packages as follows:

### Prerequisite

* Create a lakehouse to host the packages, preferably in the same workspace as your eventhouse.

### Install packages

1. For public packages in [PyPi](https://pypi.org/) or other channels,
download the package and its dependencies.

* From a cmd window in your local Windows Python environment, run:

```python
pip wheel [-w download-dir] package-name.
```

1. Create a zip file containing the required package and its dependencies.

* For private packages, zip the folder of the package and the folders of its dependencies.
* For public packages, zip the files that were downloaded in the previous step.

> [!NOTE]
>
> * Make sure to download the package that is compatible to the Python engine and the platform of the sandbox runtime (currently 3.10.8 or 3.11.7 on Windows)
> * Make sure to zip the `.whl` files themselves, and not their parent folder.
> * You can skip `.whl` files for packages that already exist with the same version in the base sandbox image.

1. Upload the zip file to the lakehouse.

1. Copy the OneLake URL (from the zipped file's properties)

1. Call the `python` plugin.
* Specify the `external_artifacts` parameter with a property bag of local name and OneLake URL of the zip file.
* In your inline python code, import `Zipackage` from `sandbox_utils` and call its `install()` method with the name of the ZIP file.

### Example using external artifacts

Install the [Faker](https://pypi.org/project/Faker/) package that generates fake data.

~~~kusto
range ID from 1 to 3 step 1
| extend Name=''
| evaluate python(typeof(*), ```if 1:
from sandbox_utils import Zipackage
Zipackage.install("Faker.zip")
from faker import Faker
fake = Faker()
result = df
for i in range(df.shape[0]):
result.loc[i, "Name"] = fake.name()
```,
external_artifacts=bag_pack('faker.zip', 'https://msit-onelake.dfs.fabric.microsoft.com/MSIT_DEMO_WS/MSIT_DEMO_LH.Lakehouse/Files/Faker.zip;impersonate'))
~~~

| ID | Name |
|----|-------|
| 1| Gary Tapia |
| 2| Emma Evans |
| 3| Ashley Bowen |

## Related content

For more examples of UDF functions that use the Python plugin, see the [Functions library](../functions-library/functions-library.md).