|
| 1 | +# Azure ADLS Storage Backend |
| 2 | + |
| 3 | +`delta-rs` offers native support for using Microsoft Azure Data Lake Storage (ADSL) as an object storage backend. |
| 4 | + |
| 5 | +You don’t need to install any extra dependencies to read/write Delta tables to S3 with engines that use `delta-rs`. You do need to configure your ADLS access credentials correctly. |
| 6 | + |
| 7 | +## Passing Credentials Explicitly |
| 8 | + |
| 9 | +You can also pass ADLS credentials to your query engine explicitly. |
| 10 | + |
| 11 | +For Polars, you would do this using the `storage_options` keyword as demonstrated above. This will forward your credentials to the `object store` library that Polars uses for cloud storage access under the hood. Read the [`object store` documentation](https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants) for more information defining specific credentials. |
| 12 | + |
| 13 | +## Example: Write Delta table to ADLS with Polars |
| 14 | + |
| 15 | +Using Polars, you can write a Delta table to ADLS directly like this: |
| 16 | + |
| 17 | +```python |
| 18 | +import polars as pl |
| 19 | + |
| 20 | +df = pl.DataFrame({"foo": [1, 2, 3, 4, 5]}) |
| 21 | + |
| 22 | +# define container name |
| 23 | +container = <container_name> |
| 24 | + |
| 25 | +# define credentials |
| 26 | +storage_options = { |
| 27 | + "ACCOUNT_NAME": <account_name>, |
| 28 | + "ACCESS_KEY": <access_key>, |
| 29 | +} |
| 30 | + |
| 31 | +# write Delta to ADLS |
| 32 | +df_pl.write_delta( |
| 33 | + f"abfs://{container}/delta_table", |
| 34 | + storage_options = storage_options |
| 35 | +) |
| 36 | +``` |
| 37 | + |
| 38 | +## Example with pandas |
| 39 | + |
| 40 | +For libraries without direct `write_delta` methods (like Pandas), you can use the `write_deltalake` function from the `deltalake` library: |
| 41 | + |
| 42 | +```python |
| 43 | +import pandas as pd |
| 44 | +from deltalake import write_deltalake |
| 45 | + |
| 46 | +df = pd.DataFrame({"foo": [1, 2, 3, 4, 5]}) |
| 47 | + |
| 48 | +write_deltalake( |
| 49 | + f"abfs://{container}/delta_table_pandas", |
| 50 | + df, |
| 51 | + storage_options=storage_options |
| 52 | +) |
| 53 | +``` |
| 54 | + |
| 55 | +## Using Local Authentication |
| 56 | + |
| 57 | +If your local session is authenticated using the Azure CLI then you can write Delta tables directly to ADLS. Read more about this in the [Azure CLI documentation](https://learn.microsoft.com/en-us/cli/azure/). |
0 commit comments