Generic OS Error when writing Delta on S3 bucket with rust engine

[montanarograziano]

Environment

Delta-rs version: deltalake package version 0.24.0

Binding:

Environment: Python 3.10.0

• Cloud provider:
• OS: MacOS Sequoia 15.2
• Other:

---

Bug

What happened:
Probabily related to #2639.
I'm trying to create a Delta Table on a S3 bucket using write_deltalake but using rust as engine causes OSError source:error sending request for url. This doesn't happen when using pyarrow.
Here's a snippet of what I'm doing. In this case I'm trying to overwrite an existing delta (due to errors) processing it in chucks as it doesn't fit in memory.

from deltalake import write_deltalake

storage_options = {
    "AWS_ACCESS_KEY_ID": os.environ.get("AWS_ACCESS_KEY_ID", ""),
    "AWS_SECRET_ACCESS_KEY": os.environ.get("AWS_SECRET_ACCESS_KEY", ""),
    "AWS_REGION": os.environ.get("AWS_DEFAULT_REGION", ""),
}
mode = "overwrite"
for year in [1970, 2023, 2024]:
    for month in range(1, 13):
        part = df.filter(pl.col("year") == year, pl.col("month") == month).collect()
        if not part.is_empty():
            print(year, month)
            write_deltalake(
                path,
                part.to_arrow(),
                mode=mode,
                storage_options=storage_options,
                engine="rust",
            )
            mode = "append"

And here's the exception:

What you expected to happen:

How to reproduce it:

More details:

[ion-elgreco]

The Rust Writer is a bit slower atm in terms of writing, storage_options = {"timeout" : "60s"} should help for example for ObjectStore to allow the requests to finish before erroring out.

[montanarograziano]

Thank you, that worked!