Skip to content

Commit a6c5dd8

Browse files
nysenthilnysenthil
committed
Committing the DPS toolkit version 1.0.8 code for the very first time
into the IBMStreams GitHub.
1 parent 96d023f commit a6c5dd8

File tree

477 files changed

+101456
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

477 files changed

+101456
-1
lines changed

‎README.md‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,22 @@
1-
# streamsx.dps
1+
The IBMStreams/streamsx.dps GitHub repository is home to the Streams toolkit named
2+
DPS (Distributed Process Store). This Streams toolkit allows a simple way for the
3+
SPL, C++ and Java operators belonging to a single or multiple applications to share
4+
the application specific state information via an external K/V store. It does this
5+
via a collection of APIs that can be called from any part of the SPL, C++ and Java
6+
operator code.
7+
8+
Following are the external NoSQL K/V stores that can be configured to work with the
9+
DPS toolkit for the purpose of sharing application state in a distributed manner.
10+
11+
1) Memcached
12+
2) Redis
13+
3) Cassandra
14+
4) IBM Cloudant
15+
5) HBase
16+
6) Mongo
17+
7) Couchbase
18+
8) Aerospike
19+
20+
There are plenty of details available about the installation, configuration, API description,
21+
built-in example etc. Please refer to the com.ibm.streamsx.dps/doc/dps-usage-tips.txt file
22+
for getting a good start in using this toolkit.

‎com.ibm.streamsx.dps/.classpath‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<classpath>
3+
<classpathentry kind="src" output="impl/java/bin" path="impl/java/src"/>
4+
<classpathentry exported="true" kind="con" path="com.ibm.streams.java/com.ibm.streams.operator"/>
5+
<classpathentry exported="true" kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
6+
<classpathentry kind="output" path="impl/java/bin"/>
7+
</classpath>

‎com.ibm.streamsx.dps/.project‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<projectDescription>
3+
<name>com.ibm.streamsx.dps</name>
4+
<comment></comment>
5+
<projects>
6+
</projects>
7+
<buildSpec>
8+
<buildCommand>
9+
<name>org.eclipse.jdt.core.javabuilder</name>
10+
<arguments>
11+
</arguments>
12+
</buildCommand>
13+
<buildCommand>
14+
<name>com.ibm.streams.studio.splproject.builder.SPLProjectBuilder</name>
15+
<arguments>
16+
</arguments>
17+
</buildCommand>
18+
<buildCommand>
19+
<name>org.eclipse.xtext.ui.shared.xtextBuilder</name>
20+
<arguments>
21+
</arguments>
22+
</buildCommand>
23+
</buildSpec>
24+
<natures>
25+
<nature>com.ibm.streams.studio.splproject.SPLProjectNature</nature>
26+
<nature>org.eclipse.xtext.ui.shared.xtextNature</nature>
27+
<nature>org.eclipse.jdt.core.javanature</nature>
28+
</natures>
29+
</projectDescription>

‎com.ibm.streamsx.dps/.settings/org.eclipse.jdt.apt.core.prefs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
eclipse.preferences.version=1
2+
org.eclipse.jdt.apt.aptEnabled=true
3+
org.eclipse.jdt.apt.genSrcDir=impl/java/src
4+
org.eclipse.jdt.apt.reconcileEnabled=false

‎com.ibm.streamsx.dps/.settings/org.eclipse.jdt.core.prefs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
eclipse.preferences.version=1
2+
org.eclipse.jdt.core.compiler.processAnnotations=enabled

‎com.ibm.streamsx.dps/.toolkitList‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
com.ibm.streamsx.dps/1.0.8
2+
spl/1.1.0
3+
058_data_sharing_between_non_fused_spl_custom_and_cpp_primitive_operators/1.0.0
4+
061_data_sharing_between_non_fused_spl_custom_operators_and_a_native_function/1.0.0
5+
062_data_sharing_between_non_fused_spl_custom_and_java_primitive_operators/1.0.0
6+
com.ibm.streams.db/1.4.0
7+
dps_test_1/1.0.0

‎com.ibm.streamsx.dps/com.ibm.streamsx/.namespace‎

Whitespace-only changes.

‎com.ibm.streamsx.dps/com.ibm.streamsx/lock/distributed/native.function/function.xml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
<functionModel
2+
xmlns="http://www.ibm.com/xmlns/prod/streams/spl/function"
3+
xmlns:cmn="http://www.ibm.com/xmlns/prod/streams/spl/common"
4+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
5+
xsi:schemaLocation="http://www.ibm.com/xmlns/prod/streams/spl/function functionModel.xsd">
6+
<functionSet>
7+
<headerFileName>DistributedLockWrappers.h</headerFileName>
8+
<functions>
9+
<function>
10+
<description>Create a new lock with a given name or get it if
11+
it already exists. Return the lock handle.</description>
12+
<prototype><![CDATA[ public stateful uint64 dlCreateOrGetLock(rstring name, mutable uint64 err) ]]></prototype>
13+
</function>
14+
<function>
15+
<description>Remove lock. Return the true if the lock was there.</description>
16+
<prototype><![CDATA[ public stateful boolean dlRemoveLock(uint64 lock, mutable uint64 err) ]]></prototype>
17+
</function>
18+
<function>
19+
<description>Acquire lock.</description>
20+
<prototype><![CDATA[ public stateful void dlAcquireLock(uint64 lock, mutable uint64 err) ]]></prototype>
21+
</function>
22+
<function>
23+
<description>Acquire lock, with an explicit lease time in seconds.</description>
24+
<prototype><![CDATA[ public stateful void dlAcquireLock(uint64 lock, float64 leaseTime, float64 maxWaitTimeToAcquireLock, mutable uint64 err) ]]></prototype>
25+
</function>
26+
<function>
27+
<description>Release lock.</description>
28+
<prototype><![CDATA[ public stateful void dlReleaseLock(uint64 lock, mutable uint64 err) ]]></prototype>
29+
</function>
30+
<function>
31+
<description>Get the process id that is currently holding a given lock.</description>
32+
<prototype><![CDATA[ public stateful uint32 dlGetPidForLock(rstring name, mutable uint64 err) ]]></prototype>
33+
</function>
34+
<function>
35+
<description>Get the description of the last DB error.</description>
36+
<prototype><![CDATA[ public stateful rstring dlGetLastDistributedLockErrorString() ]]></prototype>
37+
</function>
38+
<function>
39+
<description>Get the error code of the last DB error.</description>
40+
<prototype><![CDATA[ public stateful uint64 dlGetLastDistributedLockErrorCode() ]]></prototype>
41+
</function>
42+
</functions>
43+
<dependencies>
44+
<library>
45+
<cmn:description>Distributed process store library</cmn:description>
46+
<cmn:managedLibrary>
47+
<cmn:lib>DistributedProcessStoreLib</cmn:lib>
48+
<cmn:lib>memcached</cmn:lib>
49+
<cmn:lib>hiredis</cmn:lib>
50+
<cmn:lib>uv</cmn:lib>
51+
<cmn:lib>crypto</cmn:lib>
52+
<cmn:lib>ssl</cmn:lib>
53+
<cmn:lib>cassandra</cmn:lib>
54+
<cmn:lib>curl</cmn:lib>
55+
<cmn:lib>json-c</cmn:lib>
56+
<cmn:lib>bson</cmn:lib>
57+
<cmn:lib>mongoc</cmn:lib>
58+
<cmn:lib>couchbase</cmn:lib>
59+
<cmn:lib>lua</cmn:lib>
60+
<cmn:lib>aerospike</cmn:lib>
61+
<cmn:command>../../../../impl/bin/archLevel</cmn:command>
62+
</cmn:managedLibrary>
63+
</library>
64+
</dependencies>
65+
</functionSet>
66+
</functionModel>

‎com.ibm.streamsx.dps/com.ibm.streamsx/store/distributed/native.function/function.xml‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<functionModel xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.ibm.com/xmlns/prod/streams/spl/function" xmlns:cmn="http://www.ibm.com/xmlns/prod/streams/spl/common" xsi:schemaLocation="http://www.ibm.com/xmlns/prod/streams/spl/function functionModel.xsd">
3+
<functionSet>
4+
<headerFileName>DistributedProcessStoreWrappers.h</headerFileName>
5+
<functions>
6+
<function>
7+
<description>Create a new process store with a given name. Return the
8+
process store handle if created, 0 otherwise (a process store with the same name exists).</description>
9+
<prototype>&lt;any T1, any T2> public stateful uint64 dpsCreateStore(rstring name, T1 key, T2 value, mutable uint64 err)</prototype>
10+
</function>
11+
<function>
12+
<description>Create a new process store with a given name or get it if
13+
it already exists. Return the process store handle.</description>
14+
<prototype>&lt;any T1, any T2> public stateful uint64 dpsCreateOrGetStore(rstring name, T1 key, T2 value, mutable uint64 err)</prototype>
15+
</function>
16+
<function>
17+
<description>Find a process store given its name. Return the process
18+
store handle if found, 0 otherwise.</description>
19+
<prototype>public stateful uint64 dpsFindStore(rstring name, mutable uint64 err)</prototype>
20+
</function>
21+
<function>
22+
<description>Remove a process store given its handle. Return true if the
23+
store was there.</description>
24+
<prototype>public stateful boolean dpsRemoveStore(uint64 store, mutable uint64 err)</prototype>
25+
</function>
26+
<function>
27+
<description>Put an item into the given store. Returns true if the store did
28+
not contain an item with the same key, false otherwise.
29+
(A better performing version with no safety checks)</description>
30+
<prototype>&lt;any T1, any T2> public stateful boolean dpsPut(uint64 store, T1 key, T2 item, mutable uint64 err)</prototype>
31+
</function>
32+
<function>
33+
<description>Put an item into the given store. Returns true if the store did
34+
not contain an item with the same key, false otherwise.
35+
(A version with safety checks that will have performance overhead)</description>
36+
<prototype>&lt;any T1, any T2> public stateful boolean dpsPutSafe(uint64 store, T1 key, T2 item, mutable uint64 err)</prototype>
37+
</function>
38+
<function>
39+
<description>Put an item with a TTL (Time To Live in seconds) value into the global area of the back-end data store.
40+
Returns true if the operation is successful, false otherwise.</description>
41+
<prototype>&lt;any T1, any T2> public stateful boolean dpsPutTTL(T1 key, T2 item, uint32 ttl, mutable uint64 err)</prototype>
42+
</function>
43+
<function>
44+
<description>Get an item from the given store. Returns true if the store did
45+
contain an item with the given key and a matching type, false otherwise.
46+
(A better performing version with no safety checks)</description>
47+
<prototype>&lt;any T1, any T2> public stateful boolean dpsGet(uint64 store, T1 key, mutable T2 item, mutable uint64 err)</prototype>
48+
</function>
49+
<function>
50+
<description>Get an item from the given store. Returns true if the store did
51+
contain an item with the given key and a matching type, false otherwise.
52+
(A version with safety checks that will have performance overhead)</description>
53+
<prototype>&lt;any T1, any T2> public stateful boolean dpsGetSafe(uint64 store, T1 key, mutable T2 item, mutable uint64 err)</prototype>
54+
</function>
55+
<function>
56+
<description>Get an item that was stored with a TTL (Time To Live in seconds) value into the global area of the back-end data store.
57+
Returns true if the store did contain an item with the given key and a matching type, false otherwise.</description>
58+
<prototype>&lt;any T1, any T2> public stateful boolean dpsGetTTL(T1 key, mutable T2 item, mutable uint64 err)</prototype>
59+
</function>
60+
<function>
61+
<description>Remove an item from the given store. Returns true if the store did
62+
contain an item with the given key, false otherwise.</description>
63+
<prototype>&lt;any T1> public stateful boolean dpsRemove(uint64 store, T1 key, mutable uint64 err)</prototype>
64+
</function>
65+
<function>
66+
<description>Remove an item that was stored with a TTL (Time To Live in seconds) value into the global area of the back end data store.
67+
Returns true if an item with the given key was removed, false otherwise.</description>
68+
<prototype>&lt;any T1> public stateful boolean dpsRemoveTTL(T1 key, mutable uint64 err)</prototype>
69+
</function>
70+
<function>
71+
<description>Check if an item exists. Returns true if the given store does
72+
contain an item with the given key, false otherwise.</description>
73+
<prototype>&lt;any T1> public stateful boolean dpsHas(uint64 store, T1 key, mutable uint64 err)</prototype>
74+
</function>
75+
<function>
76+
<description>Check if an item exists that was store with a TTL (Time To Live in seconds) value.
77+
Returns true if an TTL item with the given key exists, false otherwise.</description>
78+
<prototype>&lt;any T1> public stateful boolean dpsHasTTL(T1 key, mutable uint64 err)</prototype>
79+
</function>
80+
<function>
81+
<description>Clear the given store.</description>
82+
<prototype>public stateful void dpsClear(uint64 store, mutable uint64 err)</prototype>
83+
</function>
84+
<function>
85+
<description>Get the size of the given store.</description>
86+
<prototype>public stateful uint64 dpsSize(uint64 store, mutable uint64 err)</prototype>
87+
</function>
88+
<function>
89+
<description>Begin the iteration on a given store. No other operations that can modify
90+
the state can be used until after a matching gsEndIteration()
91+
call. Returns the iterator.</description>
92+
<prototype>public stateful uint64 dpsBeginIteration(uint64 store, mutable uint64 err)</prototype>
93+
</function>
94+
<function>
95+
<description>Get the next key and value of given types in the given store. Returns true if an item was
96+
found, false otherwise.</description>
97+
<prototype>&lt;any T1, any T2> public stateful boolean dpsGetNext(uint64 store, uint64 iterator, mutable T1 key, mutable T2 value, mutable uint64 err)</prototype>
98+
</function>
99+
<function>
100+
<description>End the iteration on the given store.</description>
101+
<prototype>public stateful void dpsEndIteration(uint64 store, uint64 iterator, mutable uint64 err)</prototype>
102+
</function>
103+
<function>
104+
<description>Serialize all the items in the given store.</description>
105+
<prototype>&lt;any T1, any T2> public stateful void dpsSerialize(uint64 store, mutable blob data, T1 dummyKey, T2 dummyValue, mutable uint64 err)</prototype>
106+
</function>
107+
<function>
108+
<description>Deserialize the items from the serialized store.</description>
109+
<prototype>&lt;any T1, any T2> public stateful void dpsDeserialize(uint64 store, blob data, T1 dummyKey, T2 dummyValue, mutable uint64 err)</prototype>
110+
</function>
111+
<function>
112+
<description>Get the description of the last store error.</description>
113+
<prototype>public stateful rstring dpsGetLastStoreErrorString()</prototype>
114+
</function>
115+
<function>
116+
<description>Get the description of the last error occurred for a TTL data item.</description>
117+
<prototype>public stateful rstring dpsGetLastErrorStringTTL()</prototype>
118+
</function>
119+
<function>
120+
<description>Get the error code of the last store error.</description>
121+
<prototype>public stateful uint64 dpsGetLastStoreErrorCode()</prototype>
122+
</function>
123+
<function>
124+
<description>Get the error code of the last error ocuured for a TTL data item.</description>
125+
<prototype>public stateful uint64 dpsGetLastErrorCodeTTL()</prototype>
126+
</function>
127+
<function>
128+
<description>Get the store name for a given store id.</description>
129+
<prototype>public stateful rstring dpsGetStoreName(uint64 store)</prototype>
130+
</function>
131+
<function>
132+
<description>Get the SPl type name for the key of a given store id.</description>
133+
<prototype>public stateful rstring dpsGetSplTypeNameForKey(uint64 store)</prototype>
134+
</function>
135+
<function>
136+
<description>Get the SPl type name for the value of a given store id.</description>
137+
<prototype>public stateful rstring dpsGetSplTypeNameForValue(uint64 store)</prototype>
138+
</function>
139+
<function>
140+
<description>Get the name of the NoSQL DB product being used.</description>
141+
<prototype>public stateful rstring dpsGetNoSqlDbProductName()</prototype>
142+
</function>
143+
<function>
144+
<description>Get the name of the machine and its CPU architecture where this operator is running.</description>
145+
<prototype>public stateful void dpsGetDetailsAboutThisMachine(mutable rstring machineName, mutable rstring osVersion, mutable rstring cpuArchitecture)</prototype>
146+
</function>
147+
<function>
148+
<description>
149+
If users want to execute simple arbitrary back-end data store (fire and forget)
150+
native commands, this API can be used. This covers any Redis or Cassandra(CQL)
151+
native commands that don't have to fetch and return K/V pairs or return size of the db etc.
152+
(Insert and Delete are the more suitable ones here. However, key and value can only
153+
have string types.)
154+
</description>
155+
<prototype>public stateful boolean dpsRunDataStoreCommand(rstring cmd, mutable uint64 err)</prototype>
156+
</function>
157+
<function>
158+
<description>
159+
If users want to execute arbitrary back-end data store two way
160+
native commands, this API can be used. This is a variation of the previous API with
161+
overloaded function arguments. As of Nov/2014, this API is supported in the dps toolkit only
162+
when Cloudant or HBase NoSQL DB is used as a back-end data store. It covers any Cloudant HTTP/JSON based
163+
native commands that can perform both database and document related Cloudant APIs that are very
164+
well documented for reference on the web or it covers the HBase related REST APIs.
165+
</description>
166+
<prototype>public stateful boolean dpsRunDataStoreCommand(uint32 cmdType, rstring httpVerb,
167+
rstring baseUrl, rstring apiEndpoint, rstring queryParams, rstring jsonRequest,
168+
mutable rstring jsonResponse, mutable uint64 err)</prototype>
169+
</function>
170+
<function>
171+
<description>
172+
Base64 encode a given string. Encoded result will be returned in a
173+
user provided mutable string passed as a second function argument.
174+
</description>
175+
<prototype>public stateful void dpsBase64Encode(rstring str, mutable rstring encodedResultStr)</prototype>
176+
</function>
177+
<function>
178+
<description>
179+
Base64 decode a given string. Decoded result will be returned in a
180+
user provided mutable string passed as a second function argument.
181+
</description>
182+
<prototype>public stateful void dpsBase64Decode(rstring str, mutable rstring decodedResultStr)</prototype>
183+
</function>
184+
</functions>
185+
<dependencies>
186+
<library>
187+
<cmn:description>Distributed process store library</cmn:description>
188+
<cmn:managedLibrary>
189+
<cmn:lib>DistributedProcessStoreLib</cmn:lib>
190+
<cmn:lib>memcached</cmn:lib>
191+
<cmn:lib>hiredis</cmn:lib>
192+
<cmn:lib>uv</cmn:lib>
193+
<cmn:lib>crypto</cmn:lib>
194+
<cmn:lib>ssl</cmn:lib>
195+
<cmn:lib>cassandra</cmn:lib>
196+
<cmn:lib>curl</cmn:lib>
197+
<cmn:lib>json-c</cmn:lib>
198+
<cmn:lib>bson</cmn:lib>
199+
<cmn:lib>mongoc</cmn:lib>
200+
<cmn:lib>couchbase</cmn:lib>
201+
<cmn:lib>lua</cmn:lib>
202+
<cmn:lib>aerospike</cmn:lib>
203+
<cmn:command>../../../../impl/bin/archLevel</cmn:command>
204+
</cmn:managedLibrary>
205+
</library>
206+
</dependencies>
207+
</functionSet>
208+
</functionModel>

0 commit comments

Comments
 (0)