Skip to content

Commit fa344f7

Browse files
avriiilion-elgreco
authored andcommitted
add adls docs
1 parent 2498837 commit fa344f7

File tree

2 files changed

+59
-0
lines changed

2 files changed

+59
-0
lines changed
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Azure ADLS Storage Backend
2+
3+
`delta-rs` offers native support for using Microsoft Azure Data Lake Storage (ADSL) as an object storage backend.
4+
5+
You don’t need to install any extra dependencies to read/write Delta tables to S3 with engines that use `delta-rs`. You do need to configure your ADLS access credentials correctly.
6+
7+
## Passing Credentials Explicitly
8+
9+
You can also pass ADLS credentials to your query engine explicitly.
10+
11+
For Polars, you would do this using the `storage_options` keyword as demonstrated above. This will forward your credentials to the `object store` library that Polars uses for cloud storage access under the hood. Read the [`object store` documentation](https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants) for more information defining specific credentials.
12+
13+
## Example: Write Delta table to ADLS with Polars
14+
15+
Using Polars, you can write a Delta table to ADLS directly like this:
16+
17+
```python
18+
import polars as pl
19+
20+
df = pl.DataFrame({"foo": [1, 2, 3, 4, 5]})
21+
22+
# define container name
23+
container = <container_name>
24+
25+
# define credentials
26+
storage_options = {
27+
"ACCOUNT_NAME": <account_name>,
28+
"ACCESS_KEY": <access_key>,
29+
}
30+
31+
# write Delta to ADLS
32+
df_pl.write_delta(
33+
f"abfs://{container}/delta_table",
34+
storage_options = storage_options
35+
)
36+
```
37+
38+
## Example with pandas
39+
40+
For libraries without direct `write_delta` methods (like Pandas), you can use the `write_deltalake` function from the `deltalake` library:
41+
42+
```python
43+
import pandas as pd
44+
from deltalake import write_deltalake
45+
46+
df = pd.DataFrame({"foo": [1, 2, 3, 4, 5]})
47+
48+
write_deltalake(
49+
f"abfs://{container}/delta_table_pandas",
50+
df,
51+
storage_options=storage_options
52+
)
53+
```
54+
55+
## Using Local Authentication
56+
57+
If your local session is authenticated using the Azure CLI then you can write Delta tables directly to ADLS. Read more about this in the [Azure CLI documentation](https://learn.microsoft.com/en-us/cli/azure/).

mkdocs.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,8 @@ nav:
8282
- api/exceptions.md
8383
- Integrations:
8484
- Object Storage:
85+
- integrations/object-storage/adls.md
86+
- integrations/object-storage/gcs.md
8587
- integrations/object-storage/hdfs.md
8688
- integrations/object-storage/s3.md
8789
- integrations/object-storage/s3-like.md

0 commit comments

Comments
 (0)