support for expected-size

[vaisakh-prod]

Hi,
I came up with an issue where the expected-size needs to be updated for the boto3 client, how can I set up the same via S3fs?

--expected-size (string) This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 50GB. Failure to include this argument under these conditions may result in a failed upload due to too many parts in upload.

[martindurant]

In what context do you think you need this?

[vaisakh-prod]

I have to upload very large files (> 100 GB), and s3fs returns the following errors:

InvalidArgument: Part number must be an integer between 1 and 10000, inclusive.

[martindurant]

You are using fs.put ?

At a guess, you are using the file-like interface, and probably you need to set a bigger block-size, like fs.open(..., block_size=100*2**20).

[vaisakh-prod]

thank you, for the suggestion, I will check it out

[vaisakh-prod]

Indeed, fs.put() is used

[martindurant]

put() also takes a chunksize= parameter, default 50MB. You can change this to whatever you like.

[martindurant]

I am willing to accept a PR changing _put_file() to pass the known size of the file before starting the upload. I am not convinced this will solve the problem, however. You can experiment already, since put and put_file take arbitrary kwargs and pass them to create_multipart_upload. I don't see such an argument at https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html#API_CreateMultipartUpload_RequestParameters , though!