Spark job with AWS S3 or S3 compatible storage, s3n file size limit?

[heungheung]

Are we still using s3n:// for Spark job(s) with AWS S3 or S3 compatible storage?

If yes, is there any plan to change to s3a for AWS? How about other S3 compatible storage?

I understand there is/are adopter(s) have problem when file size is large, error will be something like

Caused by: java.lang.OutOfMemoryError: Required array size too large
 	at java.io.BufferedInputStream.fill(BufferedInputStream.java:227)
 	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
 	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
 	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
 	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
 	at org.jets3t.service.utils.ServiceUtils.hash(ServiceUtils.java:256)
 	at org.jets3t.service.utils.ServiceUtils.hashSHA256(ServiceUtils.java:290)
 	at org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:248)
 	at org.jets3t.service.impl.rest.httpclient.RestStorageService.authorizeHttpRequest(RestStorageService.java:793)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:338)
 	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:290)
 	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1175)
 	at org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:2024)
 	at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1945)
 	at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:1937)
 	at org.jets3t.service.StorageService.putObject(StorageService.java:840)
	at org.jets3t.service.S3Service.putObject(S3Service.java:2122)
 	at org.jets3t.service.S3Service.putObject(S3Service.java:2264)
 	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:122)

diving into the storeFile function, it seems to me the function storeFile
https://github.com/apache/hadoop/blob/branch-2.7.3/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore.java#L106
should auto detect whether to use storeLargeFile instead of just putObject - of course, that putObject will have size limitation.

How to enable such storeLargeFile? Is it possible to set / override this

multipartEnabled = conf.getBoolean("fs.s3n.multipart.uploads.enabled", false);

in the job level or this has been config in the whole Spark cluster?