Skip to content

Commit 8f84e98

Browse files
ShkarupaNickShkarupaNick
authored
Merge pull request #7 from elasticio/sp-159-upgrading-sailor
Sp 159 upgrading sailor
2 parents d7797cd + 4958221 commit 8f84e98

File tree

91 files changed

+7656
-677
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+7656
-677
lines changed

‎.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
*.idea
2+
/target
3+
*.iml
4+
/build
5+
/classes
6+
*.gradle
7+
out/
8+
tests.log
9+
tests.properties
10+
tests.script

‎.gitignore.save‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
out/
2+
.idea

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,4 @@ Unless required by applicable law or agreed to in writing, software
1010
distributed under the License is distributed on an "AS IS" BASIS,
1111
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1212
See the License for the specific language governing permissions and
13-
limitations under the License.
13+
limitations under the License.

‎README.md‎

Lines changed: 219 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,220 @@
1-
# jdbc
2-
Database connector for elastic.io implemented with JDBC
1+
# JDBC-component
2+
## Description
3+
This is an open source component for working with object-relational database management systems on [elastic.io platform](http://www.elastic.io "elastic.io platform").
4+
### Purpose
5+
With this component you will have following triggers:
36

4-
This project is licensed under Apache 2.0 License.
7+
``SELECT`` - this trigger will execute an [SQL](https://en.wikipedia.org/wiki/SQL "SQL") query that returns multiple results, it has limitations on the query and suited only for SELECT type of queries. The trigger will remember last execution timestamp and let you build queries on it.
8+
9+
``GET ROWS POLLING`` - this trigger will execute select query from specified table with simple criteria of selected datetime or timestamp table. The trigger will remember last execution timestamp and let you build queries on it.
10+
11+
Following actions are inside:
12+
13+
``SELECT`` - this action will execute an [SQL](https://en.wikipedia.org/wiki/SQL "SQL") query that returns multiple results, it has limitations on the query and suited only for SELECT type of queries.
14+
15+
``LOOKUP BY PRIMARY KEY`` - this action will execute select query from specified table, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns only one result (a primary key is unique).
16+
17+
``UPSERT BY PRIMARY KEY`` - this action will execute select command from specified table, as search criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"), and execute insert command by PRIMARY KEY with specified field, if result does not found, else - action will execute update command by PRIMARY KEY with specified field. The action returns only one result row (a primary key is unique).
18+
19+
``DELETE BY PRIMARY KEY`` - this action will execute delete query from specified table, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns an integer value that indicates the number of rows affected, the returned value can be 0 or 1 (a primary key is unique).
20+
### How works
21+
22+
### Requirements
23+
Before you can deploy any code into elastic.io **you must be a registered elastic.io platform user**. Please see our home page at [http://www.elastic.io](http://www.elastic.io) to learn how.
24+
#### Environment variables
25+
For unit-testing is needed to specify following environment variables:
26+
1. Connection to MSSQL:
27+
- ``CONN_USER_MSSQL`` - user login
28+
- ``CONN_PASSWORD_MSSQL`` - user password
29+
- ``CONN_DBNAME_MSSQL`` - DataBase name
30+
- ``CONN_HOST_MSSQL`` - DataBase host
31+
- ``CONN_PORT_MSSQL`` - DataBase port
32+
2. Connection to MySQL:
33+
- ``CONN_USER_MYSQL`` - user login
34+
- ``CONN_PASSWORD_MYSQL`` - user password
35+
- ``CONN_DBNAME_MYSQL`` - DataBase name
36+
- ``CONN_HOST_MYSQL`` - DataBase host
37+
- ``CONN_PORT_MYSQL`` - DataBase port
38+
3. Connection to Oracle:
39+
- ``CONN_USER_ORACLE`` - user login
40+
- ``CONN_PASSWORD_ORACLE`` - user password
41+
- ``CONN_DBNAME_ORACLE`` - DataBase name
42+
- ``CONN_HOST_ORACLE`` - DataBase host
43+
- ``CONN_PORT_ORACLE`` - DataBase port
44+
4. Connection to PostgreSQL:
45+
- ``CONN_USER_POSTGRESQL`` - user login
46+
- ``CONN_PASSWORD_POSTGRESQL`` - user password
47+
- ``CONN_DBNAME_POSTGRESQL`` - DataBase name
48+
- ``CONN_HOST_POSTGRESQL`` - DataBase host
49+
- ``CONN_PORT_POSTGRESQL`` - DataBase port
50+
#### Others
51+
## Credentials
52+
You may use following properties to configure a connection:
53+
![image](https://user-images.githubusercontent.com/40201204/43577550-ce99efe6-9654-11e8-87ed-f3e0839d618a.png)
54+
You can add the authorisation methods during the integration flow design or by going to your Settings > Security credentials > REST client and adding there.
55+
### DB Engine
56+
You are able to choose one of existing database types:
57+
![image](https://user-images.githubusercontent.com/40201204/43577772-6f85bdea-9655-11e8-96e1-368493a36c9d.png)
58+
59+
### Connection URI
60+
In the Connection URI field please provide hostname of the server, e.g. ``acme.com``
61+
### Connection port
62+
In the Connection port field please provide port of the server instance, as by default:
63+
- ``3306`` - MySQL
64+
- ``5432`` - PostgreSQL
65+
- ``1521`` - Oracle
66+
- ``1433`` - MSSQL
67+
### Database Name
68+
In the Database Name field please provide name of database at the instance that you want to interact with.
69+
### User
70+
In the User field please provide a username that has permissions to interact with the Database.
71+
### Password
72+
In the Password field please provide a password of the user that has permissions to interact with the Database.
73+
74+
Validation will start right after click on a Save button. You will be able to continue working with component after validation if all provided credentials will be valid.
75+
## Triggers
76+
### Select trigger
77+
You are able to provide SELECT query with last execution timestamp as WHERE clause criteria.
78+
![image](https://user-images.githubusercontent.com/40201204/43591075-2a032dcc-967b-11e8-968d-851355c2646e.png)
79+
Before executing the the statement %%EIO_LAST_POLL%% will be replaced with ISO Date of the last execution or max value of the last pooled datetime, for example ``2018-08-01T00:00:00.000``.
80+
During the first execution, date will be equal to ["start" of Unix Time](https://en.wikipedia.org/wiki/Unix_time) - ``1970-01-01 00:00:00.000``.
81+
Precision of the polling clause can be till milliseconds.
82+
The format of ``Start Polling From (optional)`` field should be like ``yyyy-mm-dd hh:mi:ss[.sss]``, where
83+
- ``yyyy`` - year
84+
- ``mm`` - month
85+
- ``dd`` - day
86+
- ``hh`` - hour
87+
- ``mi`` - minute
88+
- ``ss`` - second
89+
- ``sss`` - millisecond (optional)
90+
### Get Rows Polling trigger
91+
This trigger can polling data from provided table. As WHERE clause you can use column, which has datatype like DATE or TIMESTAMP.
92+
![image](https://user-images.githubusercontent.com/40201204/43591332-c99f6b3e-967b-11e8-8a77-bf8386e83d51.png)
93+
Before executing the the statement %%EIO_LAST_POLL%% will be replaced with ISO Date of the last execution or max value of the last pooled datetime, for example ``2018-08-01T00:00:00.000``.
94+
During the first execution, date will be equal to ["start" of Unix Time](https://en.wikipedia.org/wiki/Unix_time) - ``1970-01-01 00:00:00.000``.
95+
Precision of the polling clause can be till milliseconds.
96+
The format of ``Start Polling From (optional)`` field should be like ``yyyy-mm-dd hh:mi:ss[.sss]``, where
97+
- ``yyyy`` - year
98+
- ``mm`` - month
99+
- ``dd`` - day
100+
- ``hh`` - hour
101+
- ``mi`` - minute
102+
- ``ss`` - second
103+
- ``sss`` - millisecond (optional)
104+
105+
### SELECT trigger (Deprecated)
106+
This action exists in JDBC component only for backward compatibility. New [**Select trigger**](#select-trigger) is recommended to use.
107+
108+
109+
## Actions
110+
### Select action
111+
![image](https://user-images.githubusercontent.com/40201204/43592439-39ec5738-967e-11e8-8632-3655b08982d3.png)
112+
The action will execute an [SQL](https://en.wikipedia.org/wiki/SQL "SQL") query that can return multiple results, it has limitations on the query and suited only for SELECT type of queries.
113+
In SQL query you can use clause variables with specific data types.
114+
Internally we use prepared statements, so all incoming data is
115+
validated against SQL injection, however we had to build a connection from JavaScript types to the SQL data types
116+
therefore when doing a prepared statements, you would need to add ``:type`` to **each prepared statement variable**.
117+
118+
For example if you have a following SQL statement:
119+
120+
```sql
121+
SELECT
122+
FROM users
123+
WHERE userid = @id AND language = @lang
124+
```
125+
126+
you should add ``:type`` to each ``@parameter`` so your SQL query will looks like this:
127+
128+
```sql
129+
SELECT
130+
FROM users
131+
WHERE userid = @id:number AND language = @lang:string
132+
```
133+
134+
Following types are supported:
135+
* ``string``
136+
* ``number``
137+
* ``bigint``
138+
* ``boolean``
139+
* ``float``
140+
* ``date``
141+
142+
![image](https://user-images.githubusercontent.com/40201204/43644974-332f2aa4-9739-11e8-8483-f7395e5d195d.png)
143+
144+
Checkbox ``Don't throw Error on an Empty Result`` allows to emit an empty response, otherwise you will get an error on empty response.
145+
146+
#### Input fields description
147+
Component supports dynamic incoming metadata - as soon as your query is in place it will be parsed and incoming metadata will be generated accordingly.
148+
149+
### Lookup Row By Primary Key
150+
![image](https://user-images.githubusercontent.com/40201204/43592505-5b6bbfe8-967e-11e8-845e-2ce8ac707357.png)
151+
152+
The action will execute select query from a ``Table`` dropdown field, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns only one result (a primary key is unique).
153+
Checkbox ``Don't throw Error on an Empty Result`` allows to emit an empty response, otherwise you will get an error on empty response.
154+
#### Input fields description
155+
![image](https://user-images.githubusercontent.com/40201204/43644579-f593d1c8-9737-11e8-9b97-ee9e575a19f7.png)
156+
As an input metadata you will get a Primary Key field to provide the data inside as a clause value.
157+
158+
### Upsert Row By Primary Key action
159+
The action will execute ``SELECT`` command from a ``Tables`` dropdown field, as search criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"), and execute ``INSERT`` command by PRIMARY KEY with specified field, if result does not found, else - action will execute ``UPDATE`` command by PRIMARY KEY with specified field. The action returns only one result row (a primary key is unique).
160+
1. Find and select jdbc-component in the component repository
161+
![image](https://user-images.githubusercontent.com/16806832/44981615-c70a9d80-af7b-11e8-8055-3b553abe8212.png)
162+
163+
2. Create new or select existing credentials
164+
![image](https://user-images.githubusercontent.com/16806832/44981652-e86b8980-af7b-11e8-897e-04d1fc9a93cf.png)
165+
166+
3. Select action "Upsert Row By Primary Key" from list
167+
![image](https://user-images.githubusercontent.com/16806832/44981700-0d5ffc80-af7c-11e8-9ac3-aedb16e1d788.png)
168+
169+
4. Select table from ``Table`` dropdown list
170+
![image](https://user-images.githubusercontent.com/16806832/44981754-38e2e700-af7c-11e8-87d3-f029a7fec8fa.png)
171+
172+
5. Specify input data (field with red asterisk is Primary key), and click "Continue"
173+
![image](https://user-images.githubusercontent.com/16806832/44981854-83fcfa00-af7c-11e8-9ef2-8c06e77fed1e.png)
174+
175+
6. Retrieving sample
176+
![image](https://user-images.githubusercontent.com/16806832/44983059-86f9e980-af80-11e8-8178-77e463488c7a.png)
177+
178+
7. Retrieve sample result
179+
![image](https://user-images.githubusercontent.com/16806832/44982952-2ec2e780-af80-11e8-98b1-58c3adbc15b9.png)
180+
181+
8. Click "Continue"
182+
![image](https://user-images.githubusercontent.com/16806832/44983101-b0b31080-af80-11e8-82d8-0e70e4b4ff97.png)
183+
184+
9. Finish component configuration
185+
![image](https://user-images.githubusercontent.com/16806832/44983365-90378600-af81-11e8-9be4-4dbb39af0fdc.png)
186+
187+
#### Input fields description
188+
As an input metadata you will get all fields of selected table. [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY") is required field (will mark as asterisk) and other input fields are optional.
189+
![image](https://user-images.githubusercontent.com/16806832/44397461-1a76f780-a549-11e8-8247-9a6f9aa3f3b4.png)
190+
191+
### Delete Row By Primary Key action
192+
![image](https://user-images.githubusercontent.com/40201204/43592505-5b6bbfe8-967e-11e8-845e-2ce8ac707357.png)
193+
The action will execute delete query from a ``Table`` dropdown field, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns count of affected rows.
194+
Checkbox ``Don't throw Error on an Empty Result`` allows to emit an empty response, otherwise you will get an error on empty response.
195+
#### Input fields description
196+
![image](https://user-images.githubusercontent.com/40201204/43644579-f593d1c8-9737-11e8-9b97-ee9e575a19f7.png)
197+
As an input metadata you will get a Primary Key field to provide the data inside as a clause value.
198+
199+
### Create or update record action (Deprecated)
200+
This action exists in JDBC component only for backward compatibility. [**Upsert row by primary key**](#upsert-row-by-primary-key-action) Action is recommended to use.
201+
202+
## Current limitations
203+
1. Only tables with one [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY") is supported. You will see the message ``Table has not Primary Key. Should be one Primary Key
204+
``, if the selected table doesn't have a primary key. Also, you will see the message ``Composite Primary Key is not supported
205+
``, if the selected table has composite primary key.
206+
2. Only following versions of database types are supported:
207+
- ``MySQL`` - compatible with MySQL Server 5.5, 5.6, 5.7 and 8.0.
208+
- ``PostgreSQL`` - compatible with PostgreSQL 8.2 and higher
209+
- ``Oracle`` - compatible with Oracle Database 8.1.7 - 12.1.0.2
210+
- ``MSSQL`` - compatible with Microsoft SQL Server 2008 R2 and higher
211+
3. The current implementation of the action ``Upsert By Primary Key`` doesn't mark non-nullable fields as required fields at a dynamic metadata. In case of updating such fields with an empty value you will get SQL Exception ``Cannot insert the value NULL into...``. You should manually fill in all non-nullable fields with previous data, if you want to update part of columns in a row, even if data in that fields doesn't change.
212+
213+
## Known issues
214+
No known issues are there yet.
215+
216+
## License
217+
Apache-2.0 © [elastic.io GmbH](https://www.elastic.io "elastic.io GmbH")
218+
219+
## <System> API and Documentation links
220+
[elastic.io iPaaS Documentation](https://support.elastic.io/support/home "elastic.io iPaaS Documentation")

‎build.gradle‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
group = 'io.elastic'
2-
version = '2.0.2-SNAPSHOT'
2+
version = '2.3.0-SNAPSHOT'
33
apply plugin: 'java'
44
apply plugin: 'idea'
55
apply plugin: 'eclipse'
66
apply plugin: 'groovy'
77

8-
sourceCompatibility = 1.6
9-
targetCompatibility = 1.6
8+
sourceCompatibility = 1.8
9+
targetCompatibility = 1.8
1010

1111
repositories {
1212
maven {
@@ -17,18 +17,21 @@ repositories {
1717
}
1818

1919
dependencies {
20-
compile "io.elastic:sailor-jvm:1.1.2"
21-
compile "mysql:mysql-connector-java:5.1.6"
22-
compile "postgresql:postgresql:9.0-801.jdbc4"
20+
compile "io.elastic:sailor-jvm:2.1.0"
21+
compile "mysql:mysql-connector-java:8.0.11"
22+
compile group: 'org.postgresql', name: 'postgresql', version: '42.2.4'
2323
compile files("./lib/ojdbc6.jar")
2424
compile files("./lib/sqljdbc4.jar")
25+
compile group: 'com.google.code.gson', name: 'gson', version: '2.8.5'
26+
compile group: 'com.microsoft.sqlserver', name: 'mssql-jdbc', version: '6.4.0.jre8'
27+
compile 'com.fasterxml.jackson.core:jackson-core:2.7.3'
28+
compile 'com.fasterxml.jackson.core:jackson-annotations:2.7.3'
29+
compile 'com.fasterxml.jackson.core:jackson-databind:2.7.3'
2530

26-
testCompile "org.spockframework:spock-core:0.7-groovy-2.0"
27-
testCompile 'org.eclipse.jetty:jetty-server:8.1.9.v20130131'
28-
testCompile 'org.eclipse.jetty:jetty-servlet:8.1.9.v20130131'
31+
testCompile "org.spockframework:spock-core:1.1-groovy-2.4"
2932
testCompile 'org.hsqldb:hsqldb:2.0.0'
3033
}
3134

3235
task wrapper(type: Wrapper) {
33-
gradleVersion = '2.1'
36+
gradleVersion = '4.9'
3437
}

‎changelog.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# jdbc Component Change Log
2+
3+
4+
## [Unrelased]
5+
### Added
6+
### Change
7+
### Deprecated
8+
### Removed
9+
### Fixed
10+
### Security
11+
12+
## [V2.0] 2018-09-19 elastic.io
13+
14+
### Added
15+
16+
Triggers
17+
- SELECT
18+
- GET ROWS POLLING
19+
20+
Actions
21+
- SELECT
22+
- LOOKUP BY PRIMARY KEY
23+
- UPSERT BY PRIMARY KEY (for migration)
24+
- DELETE BY PRIMARY KEY
25+
26+
### Removed
27+
Actions
28+
- CreateOrUpdateRecord
29+
30+
### Fixed
31+
- fix issue in postgresql - getDate(null)
32+
- fix null values as input for select
33+

0 commit comments

Comments
 (0)