Read S3 Storage Without Object Versions

[dvnatanael]

Is it possible to use datachain.read_storage to read from an S3 bucket without the ListObjectVersions permission? I am just starting to check out DataChain, and I currently do not have this permission for the bucket in question.

The documentation on remote storage suggests that I should be able to pass {"version_aware": False} to the client_config parameter. However, the implementation of the S3 client always sets the pagination method to list_object_versions:

datachain/src/datachain/client/s3.py

Lines 104 to 114 in b57275a

	versions = True
	fs = self.fs
	await fs.set_session()
	s3 = await fs.get_s3(self.name)
	if versions:
	method = "list_object_versions"
	contents_key = "Versions"
	else:
	method = "list_objects_v2"
	contents_key = "Contents"
	pag = s3.get_paginator(method)

Finally, are there other permissions that DataChain requires which are not listed in the documentation?

Thanks in advance!

[dmpetrov]

thank you @dvnatanael!
it's disabled in many buckets by default. I've created an issue #1103

[dvnatanael]

Thank you!

[dmpetrov]

@dvnatanael it should be fixed by #1127. Could you please check it?

[dvnatanael]

I can read the storage without the ListObjectVersions permission now. Thank you!