[BUG] spark can not write to s3 due to missing storage configuration

[puchengy]

I got below error when trying to write to a lance table on s3 using spark

Caused by: java.util.concurrent.ExecutionException: java.io.IOException: LanceError(IO): failed to shutdown object writer for pyang/temp/lance_eval/0c6210c5_pyang_test$test_table_embedding/data/10111010011111010110101025256c431e9373442f85c94790.lance: Generic DynamicStorageOptionsCredentialProvider error: Missing required credentials in storage options, /workspace/rust/lance-io/src/object_writer.rs:492:17

We used AWS InstanceProfileCredentialsProvider internally, but don't see anything supported in https://lance.org/guide/object_store/#s3-configuration.

Spark version: 3.5.6
Lance Iceberg version: 0.2.0-beta.2

---

A side note: even we can not write data objects, it seems we successfully created the table (I can see metadata on s3)

2026-03-11 05:01:45          0 pyang/temp/lance_eval/0c6210c5_pyang_test$test_table_embedding/.lance-reserved
2026-03-11 05:01:48        315 pyang/temp/lance_eval/0c6210c5_pyang_test$test_table_embedding/_transactions/0-237b3b13-5b35-45f2-8c52-b6ac084025a0.txn
2026-03-11 05:01:48        717 pyang/temp/lance_eval/0c6210c5_pyang_test$test_table_embedding/_versions/18446744073709551614.manifest

which seems to indicate metadata write and data write are two different client implementations? so the client (for metadata write) don't need additional setup and the other client (for data write) need additional setup?

Per discussion: https://discord.com/channels/1437536123541782591/1440460232168833034/1481162233341018307 mark this as a bug