feat(io): add OSS storage implementation

[divinerapier]

What changes are included in this PR?

• Support AliyunOSS backend by OpenDAL
• Update doc explains why feature storage-oss is not included in storage-all.
• Fix typo

Are these changes tested?

The support for AliyunOSS is based on OpenDAL, a production-verified project. Therefore, no additional testing has been added.

Add a new example.

[liurenjie1024]

Thanks @divinerapier for this pr. But it seems that supporting oss scheme in s3 would be enough? Also there is no tests for it.

[liurenjie1024]

Do we really need this? We need it for s3 to be compatible with both s3 and s3a, is this also the case for oss?

[divinerapier]

Because data written to Iceberg via Flink/Spark + OSS shows paths starting with oss:// in the REST catalog, which cannot be parsed or read using the S3 protocol.

[liurenjie1024]

I mean why we need to keep the schema_str field for oss?

[divinerapier]

You're right, I will fix it .

[divinerapier]

    #[cfg(feature = "storage-oss")]
    Oss {
        /// uses the same client for one FileIO Storage.
        ///
        /// TODO: allow users to configure this client.
        client: reqwest::Client,
        config: Arc<OssConfig>,
    },

Fixed, PTAL @liurenjie1024

[divinerapier]

@Xuanwo PTAL 😆

I am in the process of implementing support for Aliyun OSS storage backend using OpenDAL, can you provide me with some advice on how to test this?

Thanks.

[Xuanwo]

I am in the process of implementing support for Aliyun OSS storage backend using OpenDAL, can you provide me with some advice on how to test this?

There are no good OSS-compatible services like MinIO for S3 that suit our testing needs. OpenDAL tests OSS using a user-sponsored, dedicated testing bucket. Perhaps we should consider taking a similar approach.

[divinerapier]

There are no good OSS-compatible services like MinIO for S3 that suit our testing needs. OpenDAL tests OSS using a user-sponsored, dedicated testing bucket. Perhaps we should consider taking a similar approach.

Thank you (@Xuanwo ) for raising this critical issue about OSS-compatible testing infrastructure. I fully agree that a real OSS service would be ideal for ensuring authentic testing behavior.

I previously contacted Aliyun (OSS) pre-sales support to inquire about dedicated testing environments, but unfortunately, they currently do not offer such programs for open-source projects. Given OpenDAL's successful model of using community-sponsored resources, I want to explore two potential paths:

• Community Connections:​​ Does anyone in our project community work at Aliyun(for OSS) or have access to internal testing buckets that could be securely shared?

• ​​Sponsored Testing Bucket:​​ If official support remains unavailable, would iceberg team consider co-maintaining a dedicated testing bucket?

cc @liurenjie1024

[Xuanwo]

I don't think it's a blocker.

@liurenjie1024, do you think it's a good idea to implement it first but leave it out of storage-all, so users who want to try it out have to manually enable storage-oss?

[divinerapier]

crates/examples/src/oss_backend.rs​​ provides an implementation example for loading data from Aliyun OSS.

and run it by:

cargo run --package iceberg-examples --example oss-backend --features storage-oss

[liurenjie1024]

I don't think it's a blocker.

@liurenjie1024, do you think it's a good idea to implement it first but leave it out of storage-all, so users who want to try it out have to manually enable storage-oss?

Sounds reasonable to me, but we need to add documentation to explain why we don't include it int storage-all.

[liurenjie1024]

Thanks @divinerapier for this pr, LGTM!

[Xuanwo]

Hi, @divinerapier would you like to merge with main to fix the CI?

[Xuanwo]

Thank you @divinerapier for working on this, really nice!

[Xuanwo]

I previously contacted Aliyun (OSS) pre-sales support to inquire about dedicated testing environments, but unfortunately, they currently do not offer such programs for open-source projects. Given OpenDAL's successful model of using community-sponsored resources, I want to explore two potential paths:

Would you like to submit a seperate issue for this?

[divinerapier]

An issue submitted #1188