Skip to content

Commit 2bda8cd

Browse files
author
shulkaolka
authored
Merge pull request #5 from elasticio/upsert
Upsert
2 parents c54c814 + 67d39b9 commit 2bda8cd

31 files changed

+2609
-765
lines changed

.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
out/
2-
.idea/checkstyle-idea.xml
2+
.idea

.gitignore.save

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
out/
2+
.idea

README.md

Lines changed: 74 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,25 +14,48 @@ Following actions are inside:
1414

1515
``LOOKUP BY PRIMARY KEY`` - this action will execute select query from specified table, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns only one result (a primary key is unique).
1616

17+
``UPSERT BY PRIMARY KEY`` - this action will execute select command from specified table, as search criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"), and execute insert command by PRIMARY KEY with specified field, if result does not found, else - action will execute update command by PRIMARY KEY with specified field. The action returns only one result row (a primary key is unique).
18+
1719
``DELETE BY PRIMARY KEY`` - this action will execute delete query from specified table, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns an integer value that indicates the number of rows affected, the returned value can be 0 or 1 (a primary key is unique).
1820
### How works
1921

2022
### Requirements
2123
Before you can deploy any code into elastic.io **you must be a registered elastic.io platform user**. Please see our home page at [http://www.elastic.io](http://www.elastic.io) to learn how.
2224
#### Environment variables
23-
For unit-testing
25+
For unit-testing is needed to specify following environment variables:
26+
1. Connection to MSSQL:
27+
- ``CONN_USER_MSSQL`` - user login
28+
- ``CONN_PASSWORD_MSSQL`` - user password
29+
- ``CONN_DBNAME_MSSQL`` - DataBase name
30+
- ``CONN_HOST_MSSQL`` - DataBase host
31+
- ``CONN_PORT_MSSQL`` - DataBase port
32+
2. Connection to MySQL:
33+
- ``CONN_USER_MYSQL`` - user login
34+
- ``CONN_PASSWORD_MYSQL`` - user password
35+
- ``CONN_DBNAME_MYSQL`` - DataBase name
36+
- ``CONN_HOST_MYSQL`` - DataBase host
37+
- ``CONN_PORT_MYSQL`` - DataBase port
38+
3. Connection to Oracle:
39+
- ``CONN_USER_ORACLE`` - user login
40+
- ``CONN_PASSWORD_ORACLE`` - user password
41+
- ``CONN_DBNAME_ORACLE`` - DataBase name
42+
- ``CONN_HOST_ORACLE`` - DataBase host
43+
- ``CONN_PORT_ORACLE`` - DataBase port
44+
4. Connection to PostgreSQL:
45+
- ``CONN_USER_POSTGRESQL`` - user login
46+
- ``CONN_PASSWORD_POSTGRESQL`` - user password
47+
- ``CONN_DBNAME_POSTGRESQL`` - DataBase name
48+
- ``CONN_HOST_POSTGRESQL`` - DataBase host
49+
- ``CONN_PORT_POSTGRESQL`` - DataBase port
2450
#### Others
2551
## Credentials
2652
You may use following properties to configure a connection:
2753
![image](https://user-images.githubusercontent.com/40201204/43577550-ce99efe6-9654-11e8-87ed-f3e0839d618a.png)
2854
You can add the authorisation methods during the integration flow design or by going to your Settings > Security credentials > REST client and adding there.
2955
### DB Engine
56+
You are able to choose one of existing database types:
3057
![image](https://user-images.githubusercontent.com/40201204/43577772-6f85bdea-9655-11e8-96e1-368493a36c9d.png)
31-
You are able to choose one of existing database types
32-
- ``MySQL`` - compatible with MySQL Server 5.5, 5.6, 5.7 and 8.0.
33-
- ``PostgreSQL`` - compatible with PostgreSQL 8.2 and higher
34-
- ``Oracle`` - compatible with Oracle Database 8.1.7 - 12.1.0.2
35-
- ``MSSQL`` - compatible with Microsoft SQL Server 2008 R2 and higher
58+
3659
### Connection URI
3760
In the Connection URI field please provide hostname of the server, e.g. ``acme.com``
3861
### Connection port
@@ -120,12 +143,46 @@ Component supports dynamic incoming metadata - as soon as your query is in place
120143

121144
### LOOKUP BY PRIMARY KEY
122145
![image](https://user-images.githubusercontent.com/40201204/43592505-5b6bbfe8-967e-11e8-845e-2ce8ac707357.png)
146+
123147
The action will execute select query from a ``Table`` dropdown field, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns only one result (a primary key is unique).
124148
Checkbox ``Don't throw Error on an Empty Result`` allows to emit an empty response, otherwise you will get an error on empty response.
125149
#### Input fields description
126150
![image](https://user-images.githubusercontent.com/40201204/43644579-f593d1c8-9737-11e8-9b97-ee9e575a19f7.png)
127151
As an input metadata you will get a Primary Key field to provide the data inside as a clause value.
128152

153+
### UPSERT BY PRIMARY KEY
154+
The action will execute ``SELECT`` command from a ``Tables`` dropdown field, as search criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"), and execute ``INSERT`` command by PRIMARY KEY with specified field, if result does not found, else - action will execute ``UPDATE`` command by PRIMARY KEY with specified field. The action returns only one result row (a primary key is unique).
155+
1. Find and select jdbc-component in the component repository
156+
![image](https://user-images.githubusercontent.com/16806832/44981615-c70a9d80-af7b-11e8-8055-3b553abe8212.png)
157+
158+
2. Create new or select existing credentials
159+
![image](https://user-images.githubusercontent.com/16806832/44981652-e86b8980-af7b-11e8-897e-04d1fc9a93cf.png)
160+
161+
3. Select action "Upsert Row By Primary Key" from list
162+
![image](https://user-images.githubusercontent.com/16806832/44981700-0d5ffc80-af7c-11e8-9ac3-aedb16e1d788.png)
163+
164+
4. Select table from ``Table`` dropdown list
165+
![image](https://user-images.githubusercontent.com/16806832/44981754-38e2e700-af7c-11e8-87d3-f029a7fec8fa.png)
166+
167+
5. Specify input data (field with red asterisk is Primary key), and click "Continue"
168+
![image](https://user-images.githubusercontent.com/16806832/44981854-83fcfa00-af7c-11e8-9ef2-8c06e77fed1e.png)
169+
170+
6. Retrieving sample
171+
![image](https://user-images.githubusercontent.com/16806832/44983059-86f9e980-af80-11e8-8178-77e463488c7a.png)
172+
173+
7. Retrieve sample result
174+
![image](https://user-images.githubusercontent.com/16806832/44982952-2ec2e780-af80-11e8-98b1-58c3adbc15b9.png)
175+
176+
8. Click "Continue"
177+
![image](https://user-images.githubusercontent.com/16806832/44983101-b0b31080-af80-11e8-82d8-0e70e4b4ff97.png)
178+
179+
9. Finish component configuration
180+
![image](https://user-images.githubusercontent.com/16806832/44983365-90378600-af81-11e8-9be4-4dbb39af0fdc.png)
181+
182+
#### Input fields description
183+
As an input metadata you will get all fields of selected table. [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY") is required field (will mark as asterisk) and other input fields are optional.
184+
![image](https://user-images.githubusercontent.com/16806832/44397461-1a76f780-a549-11e8-8247-9a6f9aa3f3b4.png)
185+
129186
### DELETE BY PRIMARY KEY
130187
![image](https://user-images.githubusercontent.com/40201204/43592505-5b6bbfe8-967e-11e8-845e-2ce8ac707357.png)
131188
The action will execute delete query from a ``Table`` dropdown field, as criteria can be used only [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY"). The action returns count of affected rows.
@@ -134,6 +191,17 @@ Checkbox ``Don't throw Error on an Empty Result`` allows to emit an empty respon
134191
![image](https://user-images.githubusercontent.com/40201204/43644579-f593d1c8-9737-11e8-9b97-ee9e575a19f7.png)
135192
As an input metadata you will get a Primary Key field to provide the data inside as a clause value.
136193

194+
## Current limitations
195+
1. Only tables with one [PRIMARY KEY](https://en.wikipedia.org/wiki/Primary_key "PRIMARY KEY") is supported. You will see the message ``Table has not Primary Key. Should be one Primary Key
196+
``, if the selected table doesn't have a primary key. Also, you will see the message ``Composite Primary Key is not supported
197+
``, if the selected table has composite primary key.
198+
2. Only following versions of database types are supported:
199+
- ``MySQL`` - compatible with MySQL Server 5.5, 5.6, 5.7 and 8.0.
200+
- ``PostgreSQL`` - compatible with PostgreSQL 8.2 and higher
201+
- ``Oracle`` - compatible with Oracle Database 8.1.7 - 12.1.0.2
202+
- ``MSSQL`` - compatible with Microsoft SQL Server 2008 R2 and higher
203+
3. The current implementation of the action ``Upsert By Primary Key`` doesn't mark non-nullable fields as required fields at a dynamic metadata. In case of updating such fields with an empty value you will get SQL Exception ``Cannot insert the value NULL into...``. You should manually fill in all non-nullable fields with previous data, if you want to update part of columns in a row, even if data in that fields doesn't change.
204+
137205
## Known issues
138206
No known issues are there yet.
139207

component.json

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,21 @@
124124
},
125125
"dynamicMetadata": "io.elastic.jdbc.PrimaryColumnNamesProvider"
126126
},
127+
"upsertRowByPrimaryKey": {
128+
"main": "io.elastic.jdbc.actions.UpsertRowByPrimaryKey",
129+
"title": "Upsert Row By Primary Key",
130+
"description": "Executes upsert by primary key",
131+
"fields": {
132+
"tableName": {
133+
"viewClass": "SelectView",
134+
"prompt": "Select a Table",
135+
"label": "Table",
136+
"required": true,
137+
"model": "io.elastic.jdbc.TableNameProvider"
138+
}
139+
},
140+
"dynamicMetadata": "io.elastic.jdbc.ColumnNamesWithPrimaryKeyProvider"
141+
},
127142
"deleteRowByPrimaryKey": {
128143
"main": "io.elastic.jdbc.actions.DeleteRowByPrimaryKey",
129144
"title": "Delete Row By Primary Key",

src/main/java/io/elastic/jdbc/ColumnNamesProvider.java

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
public class ColumnNamesProvider implements DynamicMetadataProvider, SelectModelProvider {
1919

20-
private static final Logger logger = LoggerFactory.getLogger(ColumnNamesProvider.class);
20+
private static final Logger LOGGER = LoggerFactory.getLogger(ColumnNamesProvider.class);
2121

2222
@Override
2323
public JsonObject getSelectModel(JsonObject configuration) {
@@ -57,8 +57,8 @@ public JsonObject getColumns(JsonObject configuration) {
5757
ResultSet rs = null;
5858
String schemaName = null;
5959
boolean isEmpty = true;
60-
Boolean isOracle = (configuration.getString("dbEngine").equals("oracle")) ? true : false;
61-
Boolean isMssql = (configuration.getString("dbEngine").equals("mssql")) ? true : false;
60+
Boolean isOracle = configuration.getString("dbEngine").equals("oracle");
61+
Boolean isMssql = configuration.getString("dbEngine").equals("mssql");
6262
try {
6363
connection = Utils.getConnection(configuration);
6464
DatabaseMetaData dbMetaData = connection.getMetaData();
@@ -71,14 +71,14 @@ public JsonObject getColumns(JsonObject configuration) {
7171
while (rs.next()) {
7272
JsonObjectBuilder field = Json.createObjectBuilder();
7373
String name = rs.getString("COLUMN_NAME");
74-
Boolean isRequired = false;
74+
Boolean isRequired;
75+
Integer isNullable = (rs.getObject("NULLABLE") != null) ? rs.getInt("NULLABLE") : 1;
7576
if (isMssql) {
7677
String isAutoincrement =
7778
(rs.getString("IS_AUTOINCREMENT") != null) ? rs.getString("IS_AUTOINCREMENT") : "";
78-
Integer isNullable = (rs.getObject("NULLABLE") != null) ? rs.getInt("NULLABLE") : 1;
7979
isRequired = isNullable == 0 && !isAutoincrement.equals("YES");
8080
} else {
81-
isRequired = false;
81+
isRequired = isNullable == 0;
8282
}
8383
field.add("required", isRequired)
8484
.add("title", name)
@@ -97,14 +97,14 @@ public JsonObject getColumns(JsonObject configuration) {
9797
try {
9898
rs.close();
9999
} catch (SQLException e) {
100-
logger.error("Failed to close result set {}", e.toString());
100+
LOGGER.error("Failed to close result set {}", e);
101101
}
102102
}
103103
if (connection != null) {
104104
try {
105105
connection.close();
106106
} catch (SQLException e) {
107-
logger.error("Failed to close connection {}", e.toString());
107+
LOGGER.error("Failed to close connection {}", e);
108108
}
109109
}
110110
}
Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
package io.elastic.jdbc;
2+
3+
import io.elastic.api.DynamicMetadataProvider;
4+
import io.elastic.api.SelectModelProvider;
5+
import java.sql.Connection;
6+
import java.sql.DatabaseMetaData;
7+
import java.sql.ResultSet;
8+
import java.sql.SQLException;
9+
import java.sql.Types;
10+
import java.util.Map;
11+
import javax.json.Json;
12+
import javax.json.JsonObject;
13+
import javax.json.JsonObjectBuilder;
14+
import javax.json.JsonValue;
15+
import org.slf4j.Logger;
16+
import org.slf4j.LoggerFactory;
17+
18+
public class ColumnNamesWithPrimaryKeyProvider implements DynamicMetadataProvider,
19+
SelectModelProvider {
20+
21+
private static final Logger LOGGER = LoggerFactory
22+
.getLogger(ColumnNamesWithPrimaryKeyProvider.class);
23+
24+
@Override
25+
public JsonObject getSelectModel(JsonObject configuration) {
26+
JsonObjectBuilder result = Json.createObjectBuilder();
27+
JsonObject properties = getColumns(configuration);
28+
for (Map.Entry<String, JsonValue> entry : properties.entrySet()) {
29+
result.add(entry.getKey(), entry.getKey());
30+
}
31+
return result.build();
32+
}
33+
34+
/**
35+
* Returns Columns list as metadata
36+
*/
37+
38+
@Override
39+
public JsonObject getMetaModel(JsonObject configuration) {
40+
JsonObjectBuilder result = Json.createObjectBuilder();
41+
JsonObjectBuilder inMetadata = Json.createObjectBuilder();
42+
JsonObjectBuilder outMetadata = Json.createObjectBuilder();
43+
JsonObject properties = getColumns(configuration);
44+
inMetadata.add("type", "object").add("properties", properties);
45+
outMetadata.add("type", "object").add("properties", properties);
46+
result.add("out", outMetadata.build()).add("in", inMetadata.build());
47+
return result.build();
48+
}
49+
50+
public JsonObject getColumns(JsonObject configuration) {
51+
if (configuration.getString("tableName") == null || configuration.getString("tableName")
52+
.isEmpty()) {
53+
throw new RuntimeException("Table name is required");
54+
}
55+
String tableName = configuration.getString("tableName");
56+
JsonObjectBuilder properties = Json.createObjectBuilder();
57+
Connection connection = null;
58+
ResultSet rs = null;
59+
ResultSet rsPrimaryKeys = null;
60+
String schemaName = null;
61+
boolean isEmpty = true;
62+
Boolean isOracle = configuration.getString("dbEngine").equals("oracle");
63+
try {
64+
connection = Utils.getConnection(configuration);
65+
DatabaseMetaData dbMetaData = connection.getMetaData();
66+
if (tableName.contains(".")) {
67+
schemaName = tableName.split("\\.")[0];
68+
tableName = tableName.split("\\.")[1];
69+
}
70+
rsPrimaryKeys = dbMetaData
71+
.getPrimaryKeys(null, ((isOracle && !schemaName.isEmpty()) ? schemaName : null),
72+
tableName);
73+
rs = dbMetaData.getColumns(null, schemaName, tableName, "%");
74+
while (rs.next()) {
75+
JsonObjectBuilder field = Json.createObjectBuilder();
76+
String name = rs.getString("COLUMN_NAME");
77+
Boolean isRequired = false;
78+
while (rsPrimaryKeys.next()) {
79+
if (rsPrimaryKeys.getString("COLUMN_NAME").equals(name)) {
80+
isRequired = true;
81+
break;
82+
}
83+
}
84+
field.add("required", isRequired)
85+
.add("title", name)
86+
.add("type", convertType(rs.getInt("DATA_TYPE")));
87+
properties.add(name, field.build());
88+
isEmpty = false;
89+
}
90+
if (isEmpty) {
91+
properties.add("empty dataset", "no columns");
92+
}
93+
94+
} catch (SQLException e) {
95+
throw new RuntimeException(e);
96+
} finally {
97+
if (rs != null) {
98+
try {
99+
rs.close();
100+
} catch (SQLException e) {
101+
LOGGER.error("Failed to close result set {}", e);
102+
}
103+
}
104+
if (rsPrimaryKeys != null) {
105+
try {
106+
rsPrimaryKeys.close();
107+
} catch (SQLException e) {
108+
LOGGER.error("Failed to close result set {}", e);
109+
}
110+
}
111+
if (connection != null) {
112+
try {
113+
connection.close();
114+
} catch (SQLException e) {
115+
LOGGER.error("Failed to close connection {}", e);
116+
}
117+
}
118+
}
119+
return properties.build();
120+
}
121+
122+
/**
123+
* Converts JDBC column type name to js type according to http://db.apache.org/ojb/docu/guides/jdbc-types.html
124+
*
125+
* @param sqlType JDBC column type
126+
* @url http://db.apache.org/ojb/docu/guides/jdbc-types.html
127+
*/
128+
private String convertType(Integer sqlType) {
129+
if (sqlType == Types.NUMERIC || sqlType == Types.DECIMAL || sqlType == Types.TINYINT
130+
|| sqlType == Types.SMALLINT || sqlType == Types.INTEGER || sqlType == Types.BIGINT
131+
|| sqlType == Types.REAL || sqlType == Types.FLOAT || sqlType == Types.DOUBLE) {
132+
return "number";
133+
}
134+
if (sqlType == Types.BIT || sqlType == Types.BOOLEAN) {
135+
return "boolean";
136+
}
137+
return "string";
138+
}
139+
}

src/main/java/io/elastic/jdbc/JdbcCredentialsVerifier.java

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,27 +10,27 @@
1010

1111
public class JdbcCredentialsVerifier implements CredentialsVerifier {
1212

13-
private static final Logger logger = LoggerFactory.getLogger(JdbcCredentialsVerifier.class);
13+
private static final Logger LOGGER = LoggerFactory.getLogger(JdbcCredentialsVerifier.class);
1414

1515
@Override
1616
public void verify(JsonObject configuration) throws InvalidCredentialsException {
1717

18-
logger.info("About to connect to database using given credentials");
18+
LOGGER.info("About to connect to database using given credentials");
1919

2020
Connection connection = null;
2121

2222
try {
2323
connection = Utils.getConnection(configuration);
24-
logger.info("Successfully connected to database. Credentials verified.");
24+
LOGGER.info("Successfully connected to database. Credentials verified.");
2525
} catch (Exception e) {
2626
throw new InvalidCredentialsException("Failed to connect to database", e);
2727
} finally {
2828
if (connection != null) {
29-
logger.info("Closing database connection");
29+
LOGGER.info("Closing database connection");
3030
try {
3131
connection.close();
3232
} catch (SQLException e) {
33-
logger.error("Failed to closed database connection", e);
33+
LOGGER.error("Failed to closed database connection", e);
3434
}
3535
}
3636
}

0 commit comments

Comments
 (0)