Skip to content

Commit ca0a4b9

Browse files
committed
update readme file
1 parent 9a93ee5 commit ca0a4b9

File tree

2 files changed

+42
-23
lines changed

2 files changed

+42
-23
lines changed

scala/README.md

Lines changed: 40 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,22 @@
11
# ibmos2spark
22

3-
The package sets Spark Hadoop configurations for connecting to
3+
The package sets Spark Hadoop configurations for connecting to
44
IBM Bluemix Object Storage and Softlayer Account Object Storage instances. This packages uses the new [stocator](https://github.com/SparkTC/stocator) driver, which implements the `swift2d` protocol, and is availble
5-
on the latest IBM Apache Spark Service instances (and through IBM Data Science Experience).
5+
on the latest IBM Apache Spark Service instances (and through IBM Data Science Experience).
66

7-
Using the `stocator` driver connects your Spark executor nodes directly
7+
Using the `stocator` driver connects your Spark executor nodes directly
88
to your data in object storage.
99
This is an optimized, high-performance method to connect Spark to your data. All IBM Apache Spark kernels
10-
are instantiated with the `stocator` driver in the Spark kernel's classpath.
11-
You can also run this locally by installing the [stocator driver](https://github.com/SparkTC/stocator)
12-
and adding it to your local Apache Spark kernel's classpath.
10+
are instantiated with the `stocator` driver in the Spark kernel's classpath.
11+
You can also run this locally by installing the [stocator driver](https://github.com/SparkTC/stocator)
12+
and adding it to your local Apache Spark kernel's classpath.
1313

1414

1515
## Installation
1616

1717
This library is cross-built on both Scala 2.10 (for Spark 1.6.0) and Scala 2.11 (for Spark 2.0.0 and greater)
1818

19-
### Releases
19+
### Releases
2020

2121
#### SBT library dependency
2222

@@ -69,8 +69,8 @@ Data Science Experience](http://datascience.ibm.com), will install the package.
6969

7070
### Snapshots
7171

72-
From time-to-time, a snapshot version may be released if fixes or new features are added.
73-
The following snipets show how to install snapshot releases.
72+
From time-to-time, a snapshot version may be released if fixes or new features are added.
73+
The following snipets show how to install snapshot releases.
7474
Replace the version number (`0.0.7`) in the following examples with the version you desire.
7575

7676
##### SBT library dependency
@@ -138,24 +138,46 @@ Add SNAPSHOT repository to pom.xml
138138
## Usage
139139

140140
The usage of this package depends on *from where* your Object Storage instance was created. This package
141-
is intended to connect to IBM's Object Storage instances obtained from Bluemix or Data Science Experience
142-
(DSX) or from a separate account on IBM Softlayer. The instructions below show how to connect to
143-
either type of instance.
141+
is intended to connect to IBM's Object Storage instances obtained from Bluemix or Data Science Experience
142+
(DSX) or from a separate account on IBM Softlayer. It also supports IBM cloud object storage (COS).
143+
The instructions below show how to connect to either type of instance.
144144

145145
The connection setup is essentially the same. But the difference for you is how you deliver the
146146
credentials. If your Object Storage was created with Bluemix/DSX, with a few clicks on the side-tab
147147
within a DSX Jupyter notebook, you can obtain your account credentials in the form of a HashMap object.
148148
If your Object Storage was created with a Softlayer account, each part of the credentials will
149-
be found as text that you can copy and paste into the example code below.
149+
be found as text that you can copy and paste into the example code below.
150+
151+
### IBM Cloud Object Storage / Data Science Experience
152+
```scala
153+
import com.ibm.ibmos2spark.CloudObjectStorage
154+
155+
// The credentials HashMap may be created for you with the
156+
// "insert to code" link in your DSX notebook.
157+
158+
var credentials = scala.collection.mutable.HashMap[String, String](
159+
"auth_url"->"https://identity.open.softlayer.com",
160+
"project_id"->"xx",
161+
"region"->"xx",
162+
"user_id"->"xx",
163+
"password"->"xx",
164+
)
165+
var bucketName = "myBucket"
166+
var objectname = "mydata"
167+
168+
var bmos = new CloudObjectStorage(sc, credentials)
169+
var rdd = sc.textFile(bmos.url(bucketName , objectname))
170+
```
171+
150172

151173
### Bluemix / Data Science Experience
152174

153175

154176
```scala
155177
import com.ibm.ibmos2spark.bluemix
156178

157-
// The credentials HashMap may be created for you with the
158-
// "insert to code" link in your DSX notebook.
179+
// The credentials HashMap may be created for you with the
180+
// "insert to code" link in your DSX notebook.
159181

160182
var credentials = scala.collection.mutable.HashMap[String, String](
161183
"auth_url"->"https://identity.open.softlayer.com",
@@ -199,7 +221,7 @@ var rdd = sc.textFile(slos.url(container , objectname))
199221
### Package Info
200222

201223
One can use the automatically generated object, `BuildInfo`, to obtain the package version
202-
and other information. This object is automatically generated by the
224+
and other information. This object is automatically generated by the
203225
[`sbt-buildinfo`](https://github.com/sbt/sbt-buildinfo) plugin.
204226

205227
```
@@ -208,9 +230,9 @@ import com.ibm.ibmos2spark.BuildInfo
208230
var buildstring = BuildInfo.toString
209231
var buildbmap = BuildInfo.toMap
210232
var buildjson = BuildInfo.toJson
211-
```
233+
```
212234

213-
## License
235+
## License
214236

215237
Copyright 2016 IBM Cloud Data Services
216238

scala/src/main/scala/Osconfig.scala

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -152,11 +152,8 @@ class bluemix(sc: SparkContext, name: String, creds: HashMap[String, String],
152152
If this value is not specified, you need to pass it when
153153
you use the url function.
154154
*
155-
* When using this from a IBM Spark service instance that
156-
* is configured to connect to particular Bluemix object store
157-
* instances, the values for these credentials can be obtained
158-
* by clicking on the 'insert to code' link just below a data
159-
* source.
155+
Warning: creating a new instance of this class would overwrite the existing
156+
spark hadoop configs if set before if used with the same spark context instance.
160157
*/
161158
class CloudObjectStorage(sc: SparkContext, credentials: HashMap[String, String], cosId: String = "") {
162159

0 commit comments

Comments
 (0)