opendalfs

opendalfs is a Python-based interface for file systems that enables interaction with different storage services by Apache OpenDAL. Through opendalfs, users can utilize fsspec's standard API to operate on all storage services supported by OpenDAL.

URL Protocols

opendalfs registers multiple fsspec protocols in the form of opendal+<service>, for example:

import fsspec

f = fsspec.open(
    "opendal+s3://my-bucket/path/to/file",
    mode="rb",
    endpoint="http://localhost:9000",
    access_key_id="minioadmin",
    secret_access_key="minioadmin",
)

The URL host is mapped to the service container (e.g. bucket for s3/gcs, container for azblob), and the URL path is used as the object key.

For other OpenDAL services, register protocols at runtime:

import opendalfs

opendalfs.register_opendal_service("oss")

Installation

Basic Installation

pip install opendalfs

Development Installation

# Install all development dependencies
pip install "opendalfs[all]"

# Or install specific groups
pip install "opendalfs[dev,test]"  # for development and testing

Development Setup

This project uses:

• Python 3.11+ for the Python interface
• ruff for code formatting and linting
• pytest for testing

For development setup and guidelines, see our Contributing Guide.

Benchmarks

The benchmark script compares Arrow direct, opendalfs (fsspec), and s3fs (fsspec) on MinIO.

uv sync --extra bench
uv run python bench/bench_read_write.py --sizes 16,32,64 --files 4 --workers 4

Configure MinIO access via OPENDAL_S3_ENDPOINT, OPENDAL_S3_BUCKET, OPENDAL_S3_ACCESS_KEY_ID, and OPENDAL_S3_SECRET_ACCESS_KEY.

You can run MinIO locally with the official binary, for example:

./minio server ./minio-data --console-address ":9001"

For profiling, you can install a tool with uv (for example py-spy) and run:

uv tool install py-spy
uv tool run py-spy record -o bench.svg -- python bench/bench_read_write.py --sizes 16,32,64 --files 4 --workers 4

High write concurrency can stall on some systems. If runs time out, reduce --fsspec-workers.

Status

See Tracking issues of 0.1.0 version for opendalfs

Contributing

opendalfs is an exciting project currently under active development. Whether you're looking to use it in your projects or contribute to its growth, there are several ways you can get involved:

• Follow the Contributing Guide to contribute
• Create new Issue for bug reports or feature requests
• Join discussions in Discussions

Getting Help

• Submit issues for bug reports
• Ask questions in discussions

License

Licensed under Apache License, Version 2.0.