-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Description
I tried to use CloudObjectStorage and connect to my private object storage, but nothing worked.
For example, the following fails:
var credentialsGilCOS = scala.collection.mutable.HashMap[String, String](
"endPoint"->"http://s3-api.us-geo.objectstorage.softlayer.net",
"accessKey"->"AAA",
"secretKey"->"AHA")
var configurationNameGil = "service"
var cosGil = new CloudObjectStorage(sc, credentialsGilCOS, configurationNameGil, "softlayer_cos")
val url = cosGil.url("gvernik", "test1812/a2/data.parquet")
peopleDF.write.format("parquet").save(url)
The following works:
sc.hadoopConfiguration.set("fs.cos.gilv.access.key","AAA")
sc.hadoopConfiguration.set("fs.cos.gilv.secret.key","AHA")
sc.hadoopConfiguration.set("fs.cos.gilv.endpoint","https://s3-api.us-geo.objectstorage.softlayer.net")
sc.hadoopConfiguration.set("fs.cos.gilv.v2.signer.type","false")
peopleDF.write.format("parquet").save("cos://gvernik.gilv/test1812/a2/data.parquet")
Metadata
Metadata
Assignees
Labels
No labels